I told you transcript changes didn’t affect grade inflation

Way back when, I blogged about a Texas proposal to include average course grades next to a student’s earned grades on the student transcript. The argument was that this could be a way to curb grade inflation. I was skeptical. 

This never came to pass in Texas, but what I didn’t know at the time that this was the practice at Cornell University.

A practice they just stopped.

It turned out that – surprise! – showing average class grades didn’t stop grade inflation. In fact, showing class averages probably increased grade inflation. Because with easy access to average course grades, students preferentially took the classes seen to be “easy A’s”.

I have to admit I didn’t see that possibility, but it tracks.

Related posts

The “Texas transcript” is a good idea, but won’t solve grade inflation

External links

Cornell Discontinues Median Grade Visibility on Transcripts 15 Years After Inception  

Rats, responsibility, and reputations in research, or: That generative AI figure of rat “dck”

Say what you will about social media, it is a very revealing way to learn what your colleagues think.

Last week, science Twitter could not stop talking about this figure:

There were two more multi-part figures that are less obviously comical but equally absurd.

The paper these figures were in has now been retracted, but I found the one above in this tweet by CJ Houldcroft. You can also find them in Elizabeth Bik’s blog.

This is clearly a “cascade of fail” situation with lots of blame to go around. But the discussion made me wonder where people put responsibility for this. I ran a poll on Twitter and a poll on Mastodon asking who came out looking the worst. The combined results from 117 respondents were:

Publisher: 31.6%
Editor: 30.8%
Peer reviewers: 25.6%
Authors: 12.0% 

I can both understand these results to some degree and have these results blow my mind 🤯 a little. 

People know the name of the publisher, and many folks have been criticizing Frontiers as a publisher for a while. Critics will see this as more confirmation that Frontiers is performing poorly. So Frontiers looks bad.

The editor and peer reviewers look bad because, as the saying goes, “You had one job.” They are supposed to be responsible for quality control and they didn’t do that. (Though one reviewer sad he didn’t think the figures were his problem, which will get its own post over on the Better Posters blog later.)

But I am still surprised that the authors are getting off so lightly in this discussion. It almost feels like blaming the fire department instead of the arsonist.

At the surface level, the authors did nothing technically wrong. The journal allows AI figures if they are disclosed, and the authors disclosed it. But the figures are so horribly and obviously wrong that to even submit it feels to me more like misconduct than sloppiness.

And is so often the case, when you pull at one end of a thread, it’s interesting to see what starts to unravel.

Last author Ding-Jun Hao (whose name also appears in papers as Dingjun Hao) has had multiple papers retracted before this one (read PubPeer comments on one retracted paper), which a pseudonymous commenter on Twitter claimed was the work of a papermill. Said commenter further claimed that another paper is from a different papermill.

Lead author Xinyu Guo appears to have been author on another retracted paper.

I’ve been reminded of this quote from a former journal editor:

“Don’t blame the journal for a bad paper. Don’t blame the editor for a bad paper. Don’t blame the reviewers for a bad paper. Blame the authors for having the temerity to put up bad research for publication.” - Fred Schram in 2011, then editor of Journal of Crustacean Biology

Why do people think the authors don’t look so bad in this fiasco?

I wonder if other working scientists relate all to well to the pressure to publish, and think, “Who among us has not been tempted to use shortcuts like generative AI to get more papers out?”

I wonder if people think, “They’re from China, and China has a problem with academic misconduct.” Here’s an article from nine years ago about China trying to control its academic misconduct issues.

I wonder if people just go, “Never heard of them.” Hard to damage your reputation if you don’t have one.

But this strategy may finally be too risky. China has announced new measures to improve academic integrity issues, which could include any retracted paper requiring an explanation. And the penalties listed could be severe. Previous investigations of retractions in China resulted in “salary cuts, withdrawal of bonuses, demotions and timed suspensions from applying for research grants and rewards.”

Related posts

The Crustacean Society 2011: Day 3

References

[Retracted] Guo X, Dong L, Hao D, 2024. Cellular functions of spermatogonial stem cells in relation to JAK/STAT signaling pathway. Frontiers in Cell and Developmental Biology 11:1339390. https://doi.org/10.3389/fcell.2023.1339390

Retraction notice for Guo et al.

External links

Scientific journal publishes AI-generated rat with gigantic penis in worrying incident

Study featuring AI-generated giant rat penis retracted, journal apologizes 

The rat with the big balls and the enormous penis – how Frontiers published a paper with botched AI-generated images

China conducts first nationwide review of retractions and research misconduct (2024)

China pursues fraudsters in science publishing (2015)

Reformscape is here!

Since I joined DORA, the biggest project I have been working on pretty much nonstop is now ready. It’s called Reformscape.

Literally, Reformscape is a searchable database of university policies around research assessment.

Figuratively, it’s a tool to help you get unstuck. 

Having been down the road of trying to make policy changes at the department level myself (and not getting very far), I know that it can be hard to convince people that this is worth doing and that easy to get discouraged. By collecting examples of these institutional reforms, I hope that someone who was in my position can look at examples in Reformscape and believe, “Change is possible.”

Here’s the press release for Reformscape.

Here’s a tweetorial showing some of the features in Reformscape.

And here’s a YouTube version of the Reformscape promo video above.

I wasn’t involved in the concept or start of this project, but I was very pleased to be able to help it across the finish line. 🏁 

I’ll tell you just one little thing that I am stupidly proud of. When you listen to the Reformscape video, you might notice that the voice isn’t speaking in a typical American accent, or a typical British accent. Because DORA is a global organization, I thought it was very important that the voice on the video reflect that. Our voiceover actor brought a desi accent, and I love that to pieces.

External links

Reformscape

A JIF myth

“Why do zebras have stripes?”

“Why did T. rex have such small arms?”

In evolutionary biology, functional questions like those are notoriously tricky to answer, because people tend to mix up two separate questions: the origin of the feature, and the current use of the feature.

People often have difficulty grasping that those two things are different. This leads to questions like, “What good is half a wing?” . The implication is that that because wings are used for flying now, they must always have been used for flying, so how could they evolve?

The answer is that the bits that make up a wing can be used for lots of things, and that you can have a functional shift. Something that’s great for insulation or display like feathers proves incidentally useful for gliding. The incidental use eventually becomes the primary use.

I mention this because I stumbled across a myth about the Journal Impact Factor™️. You will find statements here and there Eugene Garfield created the Impact Factor to help libraries decide what journals to buy. This was back in the day when journal subscriptions were sold individually rather than in bundled “big deals.”

But Garfield’s own account of the history of the Impact Factor shows this is not true (link to edited version; full version can be found in external links).

Garfield was involved in creating the Genetics Citation Index in the 1950s, and needed to decided what journal to include in the index. They first tried just counting citations to journals, which favoured journals that just published a lot of papers. They realized that “total citations” missed journals that published fewer papers that were highly cited.

Impact Factor helped solve this problem. It wasn’t about libraries at all. So where did the “Impact Factor was created for libraries” belief come from?

In the 1960s, the Genetics Citation Index broadened out to become the Science Citation Index. Garfield did lots of research on citation patterns, and published a 1972 paper in Science about citation analysis. Garfield talks about all the findings using this measure, Impact Factor, that he created more than a decade earlier.

But at the end of the paper, there’s a section that begins, “Some applications,” and the first sentence is:

The results of this type of citation analysis would appear to be of great potential value in the management of library journal collections.

And that’s the origin of the myth that Impact Factor was created for libraries.

That 1972 Science paper went on to become the one that many people referred to in discussions of Impact Factor. And it’s easy to see how Garfield’s suggestion that libraries could use Impact Factor to make purchasing decisions could morph into, “Garfield created Impact Factor for libraries.” Because people don’t always read the original papers of the things they cite in details.

The Impact Factor not only shows how easily origin gets muddled, it also shows the concept of functional shift. Because libraries did use it for purchasing decisions, and then administrators started using it to make hiring, tenure, and promotion decisions. DORA is trying to provide a new selection pressure that could cause another functional shift, away from evaluating researchers.

External links

Essays about Impact Factor by Eugene Garfield

Garfield E. 1972. Citation analysis as a tool in journal evaluation. Science 178: 471-479. https://doi.org/10.1126/science.178.4060.471 

Anonymous Predatory Reports site is predatory

Some time ago, I wrote about the anonymous website Predatory Reports. Some things about it struck me as sketchy. 

It looks like my instincts were on the money. 💵 

The commercial journal vetting service Cabells is reporting major misdeeds from Predatory Reports. Specifically that:

1. The site has a bunch of plagiarized material.
2. The site owner tried to squeeze money out of Cabells.

I doubt this story is done. This is, I think, only the second act. Since the story isn’t done. I don’t know if it’s appropriate to have a moral quite yet, but it might be one of the academic Moscow rules: always trust your gut.

Hat tip to Anna Abalkina.

External links

Unmasking a predator: PredatoryReports.org

Related posts

A star chamber for predatory journals

Kung Fu Panda is going to ruin my life

I’m at the movies last night, watching the previews, and up comes a blurb about Kung Fu Panda 4.

Now I have to tell you, I love the Kung Fu Panda movies. So I’m excited!

And the presenter says, “We know there’s going to be a new character called ‘Zhen.’”  (The ‘h’ is silent, at least as pronounced by the presenter.)

And as you can see, this character is a fox.

Zhen the fox.

Zhen. Fox.

I must now accept that this is what people are going to think of when they hear my name for years.

I suppose it’ll make for a change from motorcycle repair jokes, though.

Cell Bio 2023

How it started:

How it’s going:

Second photo from https://x.com/lenakumba/status/1731423282228867322

26 December 2023: Originally posted on 3 December 2034 on the Better Posters blog by accident.

A biologist in a physics mag

The change in my job is slowly starting to have an effect. I’m quoted, as DORA Program Director, in a new article in Physics Today.

As a biologist, I have to say that this is not ever something I aspired or expected in my career. So, that’s interesting.

References

Feder T. 2023. Global movement to reform researcher assessment gains traction. Physics Today 76(10): 22–24. https://doi.org/10.1063/PT.3.5323

Social media for scientists, circa summer 2023

Twitter is now X and its credibility is shattered. Even though a lot of people keep using it, but it’s the grudging use of people who are mostly only doing it because they have to.

So both personally and professionally, I’ve been looking at other social media platforms. In my view, only two are serious contenders for filling the void left by the gross mismanagement of Twitter.

BlueSky is the new science Twitter. It’s filled with the early adopters of Twitter, the Internet experimenters and science communicators. T-shirt and jeans. And the vibe is very much the freewheeling, ever-so-slightly – okay, maybe more than slightly – anarchic vibe of the early science online days.

Mastodon is  the new Academic Twitter. (The use of capitals is deliberate.) It’s got the feel of a lot of the late adopters of Twitter. The ones who were sort of persuaded that they could use a social media platform social media for promoting their brand and posting critiques of papers and doing, you know, being a serious scholar online. Polo shirts and slacks. The vibe is ever-so-slight slightly – okay, maybe more than slightly – po-faced and straight-laced.

Now, of course, your mileage may vary. These are my own personal impressions, not back up by any data. And to be clear, I am not saying that either is better than the other. There are clearly communities developing on both platforms that are finding worthwhile discussions.

External links

Thousands of scientists are cutting back on Twitter, seeding angst and uncertainty

Space: 1999 is now further in the past than it was set in the future

Space: 1999 first aired on October 17, 1975. Unusually for a science fiction show, it gave a quite precise date for when its events started unfolding.

The first episode, “Breakaway,” is set on September 13, 1999. (September 13 is now known as “Breakaway Day” among series fans.)

That means, according to a handy duration calculator, that the first episode of Space: 1999 was set 23 years, 10 months, 27 days in the future. Or 8,732 days, if you prefer.

Today marks 8,732 days since September 13, 1999. 

The show is now officially more retro than it was ever futuristic.

Man, I’m old. But I still love the Eagle transporters, though.

Malaysia puts pressure on researcher’s publication choices

Previously, I organized a roundtable discussion for the tenth anniversary of DORA about institutions trying to “put a thumb on the scale” of researchers’ decisions about what journal to publish in. 

For brevity, I called these “bans.” But I know that in many cases, these were not true “bans.” Researchers could still publish in journals from particular publishers, but there was no doubt that the intention was to get researchers not to use particular venues.

Little did I know that an entire nation would use the power of the state in this way.

With some optical character recognition and Google Translate, the letter appears to say roughly this:

CHIEF DIRECTOR OF HIGHER EDUCATION
DIRECTOR GENERAL OF HIGHER EDUCATION
Department of Higher Education
Ministry of Higher Education Malaysia 

Date 13 July 2023

PROHIBITION OF USING GOVERNMENT FUNDS TO PUBLISH IN THREE INTERNATIONAL PUBLISHERS

With all due respect I refer to the matter at the base.

2. For the information of YBhg. Tan Sri/Datuk/DatolProf., Post-Cabinet Meeting of the Ministry of Higher Education No. 9/2023 discussed the issue of the involvement of academics who publish articles in predatory journals and poor quality journals. In this regard, the Ministry is very serious about the aspect of academic integrity and authority in research and publication activities. One (1) special committee has been created by the Ministry as an effort to control the publication of poor quality journals in order to improve the level of academic ethics, and to protect the good name of IPT and the country.

3. Therefore, the Ministry does not allow Public Universities to use the Public Fund allocation for any publication published by three (3) publishers namely Hindawi, Frontier and MDPI with immediate effect. The Ministry also advises that each University can play a role in monitoring issues related to the quality of publications more carefully in the future.

4. The cooperation and attention of YBhg. Tan Sri/Datuk/Dato7Prof. regarding this matter is greatly appreciated and preceded by a word of thanks.

"MALAYSIA MADAM" "SERVING THE COUNTRY"

I am the one who trusts,

PROF. DR. Azlinda Binto Azman

Needless to say, this made me think the roundtable was a good idea. And that these policies are problematic. The basis for this decision is completely opaque – at least in this letter. Maybe there are some more background on the Higher Education ministry website, but I haven’t had a chance to search and apply Google Translate to find out if it exists or not.

Hat tip to Mohammed Al-Amr for posting this.

Related posts

“Publisher bans & DORA” roundtable recording and notes

Social media update

There has been a proliferation of sites trying to take advantage of disaffected users leaving Twitter.

I have been creating  accounts, of course, mostly to keep anyone from grabbing my preferred handle. Here’s the update of where you can find me.

• Bluesky: @doctorzen.bsky.social
  
• Mastadon: @DoctorZen@scholar.social
• Threads: doctorzen.threads.net

Will any of these survive? Or will they go the way of Google Plus?

 

Professional news: I’m now at DORA!

So, uh, I’ve got a new job. And it6s is the first time an organization has sent out an announcement of my hire.

It’s going to be a big new challenge. Lucky for me I love big new challenges!

External links

A warm welcome to DORA’s new Program Director Zen Faulkes

Twitter owner cures millions of social media addiction

The title of this post is the joke I would normally make on Twitter. 

Except now the number of tweets people can read is being limited and I don’t want to use anyone’s reading limit up unnecessarily,

https://www.theverge.com/2023/7/1/23781198/twitter-daily-reading-limit-elon-musk-verified-paywall 

https://news.sky.com/story/twitter-applies-temporary-reading-limits-for-all-users-elon-musk-announces-12913108

The reason given for imposing a reading limit sounds weird. But that subscribers get to read 100 times more tweets than everyone else feels like a dumb ploy to get more people to pay for Twitter verification.

I keep waiting to get invitations to Bluesky, which seems to be the best candidate to take over as the major social media platform. And I am glad that I never stopped blogging.

Edit, 2 July 2022: Apparently the reading limits are a direct result of the owner of Twitter not paying the bills. That they were going to stop payments to Google Cloud was reported weeks ago.

The Zen of Presentations, Part 74: Design your slides for your audience, not social media

 Recently I spotted this conference slide shared on Twitter.

A slide with no visuals, just a wide screen of four paragraphs of text.

I thought we had all agreed not to do these any more?

At first I thought this was just crazy, but then I realized there might (and I stress might) be a something else at play here: social media. Typically, a good slide for a presentation is visual and had minimal text. The speaker provides the words, not the slide.

But a disadvantage of slides that are well designed for the audience is that they don’t make much sense without the speaker. They lack context. They aren’t portable.

This slide is the conference equivalent of someone taking screenshots of newspaper articles to get around Twitter’s character limits. Sure, the slide sucks for the audience in the room, but it’s great if you share it on Twitter, since the slide leaves no doubt exactly what the person’s point is.

I’m now very nervous that we’re going to start seeing conference slides backsliding into the old bad habits of “death by PowerPoint¨ with huge blocks of text that people will read, word for word, with the excuse being that “more people will share it on socials.”

Please make your slides for the audience in the room with you. It’s respectful. You want to share something on social media? Design something specifically for Twitter or INstagram or Bluesky or whatever.

DORA roundtable round-up and new crayfish preprint

I have two new things to read (if you’re so inclined).

First, I have a summary of my last month’s “Publisher bans & DORA” roundtable on the DORA website.

Second, I have a new preprint on regulating crayfish as pets. Saskatchewan banned a couple of crayfish species. Did passing a law do what it was supposed to - change the availability of the crayfish and make them less likely to be introduced? Maybe yes? 

This is currently in review, so I hope that there will be an official version of record later. I’ll maybe talk more about the backstory if I can get this into a journal.

External links

Publisher bans & DORA guest blog post

Online advertisements for crayfish decrease after a provincial ban

First women’s footy game!

I became a card-carrying member of an AFLW team in season 1. I was finally pleased to see a women’s footy game in person – and a championship game, no less!

Over the weekend, Hamilton hosted the AFL Canada Cup, a tournament for North American women’s teams. I didn’t get to see the whole tournament, but I did make it for the final game between the Calgary Kookaburras (actually joint with Burnaby Eagles) and the Etobicoke Kangaroos.

Unfortunately, things did not go well for the Kangaroos. And this wasn’t the final score, which I think was 50 to zero? Weirdly, I can’t find any definitive report of game results in the tournament.

External links

The AFL Canada Cup

Aussie-rules football Canadian-style

AFL Canada Instagram

The “blog and secretaries” model of scholarly publishing

Steve Harnad has done much for open access. So I was surprised by a new article, in which he wrote:

In the old pre-digital days of S&S (“scientists and scholars” - ZF) publishing, the true costs of providing print-on-paper to would-be users required the services of another profession for the production and delivery. But (let’s cut to the quick) those days are over, forever. Online publication is not altogether cost-free, but the costs are so ridiculously low that all an S&S author needs pay for is a blog service-provider, rather like a phone or email service provider. ...

All other goods and services are now obsolete – or almost obsolete: scaled down to only the tiny, trivial cost per article of managing the peer review, by paying secretaries to run the software for soliciting and monitoring it.

This is such misleading characterization of what is required to run an online publication that I kind of can’t believe he wrote it. I think he knows better – I hope e knows better – but is oversimplifying to try to make a point.

Before I go on, I want to be clear. I am not about to argue here is not that journal prices are fair, or they could not be lower. When I commented on this article on Twitter, Björn Brembs came in and provided what the article lacked: some real values associated with costs. 

What I want to comment on is Harnad’s characterization of the infrastructure of an online publishing platform. A blog service and a secretary is pretty much all you need, according to him.

In the last decade, there have been no shortage of people providing online scholarly platforms. Many of those are trying hard to move away from legacy academic publishing and the costs associated with those traditional journals.

Very few have been able to make the “blog and secretary” model work. There are possible exceptions, judging from this description of the Journal of Machine Learning Research. Keep in mind, though, that this description is now over ten years old. I am sure there are other niches for the “blog and secretaries” model, but I don’t think this is the way forward for all scholarly communication.

Preprint servers provide open access for readers without fee to authors. They are no legacy publishers, and would seem to be prime candidates for Harnad’s proposed ultra-lightweight publishing model.

This article describes how several preprint servers were struggling to get enough money to run. Since the article appeared in 2020, a of them – INA-Rxiv, ArabiXiv – are no longer taking new submissions.

Brembs mentioned Humanities Common as a model. It has expenses of about US$750,000 and is run by three major committees. This still seems a far cry from the picture Harnad paints of using off the shelf software and a skeleton staff.

Likewise, larger preprint servers have higher costs and a bureaucracy to sustain them. The big one, arXiv has a budget of US$4.8 million.

I would love to survey of researchers on their estimates of the costs of a preprint server and how many people it takes to keep them running. Prediction: I bet a lot of them would underestimate the cost and time it takes to do so, perhaps influenced by descriptions like Harnad’s.

External links

Publishers can’t be blamed for clinging to the golden goose

Popular preprint servers face closure because of money troubles

“Publisher bans & DORA” roundtable recording and notes

 Today, I was the moderator for the “Publisher bans & DORA” roundtable! Everything worked as it was supposed to, and we kept the discussion to a tight hour.

Thanks to my panelists Payal Joshi, Katherine Stephan, and Jennifer Coston-Guarini for joining me and keeping the discussion lively and interesting!

A video of the roundtable is on YouTube. I edited the video, but edits may not be processed by YouTube yet. If you see "dead air" at the start of the video, jump to 7:29.

I have saved some material on Figshare. Currently, it’s just the introductory material about events that inspired the roundtable. It may expand in the coming days.

External links

Recording of “Publisher bans & DORA” on YouTube

Notes and other material from “Publisher bans & DORA”

I told you the Lemon test was important

I’ve been expecting new bills in the US that re-introduce “intelligent design” into schools. Reason? The Kennedy v. Bremeton School District ruling in 2022. 

Surprisingly, a lot of people do not seem to think this is likely.

But state legislators sure noticed the court case.

You may have heard that there was a bill introduced into the Texas legislature to require schools to display the Ten Commandments of the Bible (88R SB1515). It’s passed the Senate.

The analysis of the bill includes a section on the intent of the bill. And it’s super clear: the bill is being pushed because the Kennedy ruling said the Supreme Court was abandoning the Lemon test.

This legislation only became legally feasible with the United States Supreme Court's opinion last year in Kennedy v. Bremerton School District, 142 S. Ct. 2407 (2022), which overturned the Lemon test under the Establishment Clause (found in Lemon v. Kurtzman, 403 U.S. 602 (1971)) and instead provided a test of whether a governmental display of religious content comports with America's history and tradition.

If the loss of the Lemon test was an open invitation for lawmakers to display religion in schools, why on earth would they stop with displays? Why would you not start to push for religious instructions in public schools?