09 January 2025

We were friends once

In 2021, I was in front of the US Consulate in Toronto. And there was a small plaque that was surprisingly moving to me.


As the United States marks the 10th anniversary of the terror attacks of September 11, 2001, in the name of all Americans, Consulate General Toronto thanks the people of Ontario for their support and generosity following the worst attack on American since WWII.

The nearly 3,000 innocent victims from ninety countries included twenty-four Canadians, our friends and family in Ontario, and all of Canada, were with us in our darkest hour, as you always are. Neither terrorism nor any adversity can conquer free peoples, and we are grateful to stand with the best neighbours any nation ever had.

As together we look back, together we go forward.

We are eternally grateful.

September 11, 2011

 

Reading the text of this now is just crushing to me. I’m sad that “eternal” gratitude seems to have lasted less than 15 years since the plaque was revealed.

External links

Thanks and remembrance

08 January 2025

More publishers toying with AI in journal review

The Bookseller is reporting that Springer Nature is testing AI checks in their production pipeline for journal articles.

According to the publisher, the AI systems are designed to alert editors to “quality issues.” I find this frustratingly unclear, much like Elsevier’s testing of production pipeline changes. What, exactly, are they looking for with these systems?

I can see some automatic checking being very valuable. For example, it blows my mind that many journals apparently do no basic checks for plagiarism. I can also see value in an AI system that scanned basic statistical information, like whether reported p values were possible given the sample size, test statistic, and so on. For example, there is a program called Statcheck that has been proposed for exactly such purposes (Nuijten & Wicherts 2023, 2024), although there are ongoing debates about its utility (Schmidt 2017, Böschen 2024).

If the publishers were confident that these systems were genuinely making the peer review process better and were catching things like:

  • Image manipulation
  • Plagiarism
  • Tortured phrases
  • Citation cartels
  • Non-existent authors or reviewers
  • Statistical errors 
  • Undisclosed use of generative AI

All of which are real problems that need addressing. So why are academic publishers being so cagey about what processes they are implementing and what they are supposed to catch? Are they worried that this provides information that cheaters can use to bypass their “quality issues” detection systems? Something else?

Publishers always claim that they add value to academic publication. These new AI checks provide a real opportunity that they could show how they are adding value to academics who are increasingly mad at them and asking “What good are you?”

Related posts

Elsevier turns generative AI loose on manuscripts for no discernable reason

External links

Springer Nature reveals AI-driven tool to ‘automate some editorial quality checks’

References

Böschen I. 2024. statcheck is flawed by design and no valid spell checker for statistical results. arXiv: 2408.07948. https://doi.org/10.48550/arXiv.2408.07948 

Nuijten MB, Wicherts J. 2023. The effectiveness of implementing statcheck in the peer review process to avoid statistical reporting errors. PsyArXiv. https://doi.org/10.31234/osf.io/bxau9

Nuijten MB, Wicherts JM. 2024. Implementing Statcheck during peer review is related to a steep decline in statistical-reporting inconsistencies. Advances in Methods and Practices in Psychological Science 7(2): 25152459241258945. https://doi.org/10.1177/25152459241258945
 
Schmidt T. 2017. Statcheck does not work: All the numbers. Reply to Nuijten et al. (2017). PsyArXiv. https://psyarxiv.com/hr6qy

31 December 2024

Elsevier turns generative AI loose on manuscripts for no discernable reason

From time to time, editorial boards quit journals. It happens often enough that I usually don’t pay much attention. But the outgoing editors of the Journal of Human Evolution described a new complaint, involving – surprise! – generative AI.

In fall of 2023, for example, without consulting or informing the editors, Elsevier initiated the use of AI during production, creating article proofs devoid of capitalization of all proper nouns (e.g., formally recognized epochs, site names, countries, cities, genera, etc.) as well italics for genera and species. These AI changes reversed the accepted versions of papers that had already been properly formatted by the handling editors. This was highly embarrassing for the journal and resolution took six months and was achieved only through the persistent efforts of the editors. AI processing continues to be used and regularly reformats submitted manuscripts to change meaning and formatting and require extensive author and editor oversight during proof stage. 

This is maddening, because it’s yet another example of gen AI creating problems, never solving them.

I am also baffled. Because usually I can at least understand why a publisher has done certain things. Often the explanation tracks back to, “Cut costs.”

But I cannot for the life of me figure out why a publisher would let a generative AI system loose on a completed, edited manuscript. I cannot believe that is in any way cost-saving.

Generative AI is notoriously expensive. And they were doing this to nominally completed manuscripts, added an additional layer of work. If it were in the hopes of saving costs eventually, you would think there would just be some internal testing, not being unleashed like a rabid raccoon on a working journal.

Nor do I think anyone with any publishing experience would believe it would improve the manuscript.

I am left absolutely confused by what Elsevier is thinking here. But moreover, I am am worried that other publishers are going to try the same thing and we just haven’t heard about it yet.

Other major publishers should see an opening here. They could get out and publicly promise academics that the job of final edit stays with human editors.

Unfortunately, I am having a hard time seeing this happening, as I suspect increasingly the heads of these companies see themselves as data and analytics companies more than publishing companies.

Update, 6 January 2025: Retraction Watch has a response from Elsevier to the resignation. They claim they weren’t using generative AI, but admit they “trialled a production workflow” that caused the errors.

I’m not sure that’s much better.

Obviously, you also hope that workflow changes would be well-thought out enough that they wouldn’t introduce mistakes.

But I’m more unimpressed by Elsevier’s lack of transparency. What were they doing to the workflow that caused the mistakes? That the editors thought this was generative AI suggests that Elsevier did not explain the new workflow to the editors well, if at all. 

Update, 10 January 2025: Science magazine is covering this story, and they have quotes from the editors that contradicts Elsevier’s “We weren’t using AI” claim:

According to Taylor and paleoanthropologist Clément Zanolli at the University of Bordeaux, another former editor-in-chief who signed the statement, Elsevier told them in 2023 that AI software had introduced the formatting errors.

Elsevier told Science the same thing they told Retraction Watch: they were testing a new production system. But the statement to Science did not say there was no AI involved, which they said to Retraction Watch.

External links

Evolution journal editors resign en masse to protest Elsevier changes

Evolution journal editors resign en masse 

Elsevier denies AI use in response to evolution journal board resignations 

Journal editors’ mass resignation marks ‘sad day for paleoanthropology’

20 December 2024

Le Monde est à vous: Academic hoaxes in French newspaper article

The second article that arose from my posting of preprints is now available. The title is, “Why scientific hoaxes aren’t always useless.”

Translation of first paragraph by Google Translate:

Canadian biologist Zen Faulkes is not a naturalist, even if he likes to collect crayfish or sand crabs for his research. However, he has a taste for unexpected collections. For several years, he has been collecting, listing and classifying... scientific hoaxes. That is to say, parody, ironic or insane research articles that should never have been published. “Of course, if these texts disappeared, it would not be a great loss. But it is important to keep track of them and try to learn some lessons from their existence,” says the researcher, whose collection spans 432 pages, for forty-two examples.

While I have been very pleased in the last few weeks to have been quoted in a couple of fairly high profile venues, I am struck by the disconnect this week between good professional news and terrible personal news. (There have been a couple of deaths close to our family in the last week.)

External links

Pourquoi les canulars scientifiques ne sont pas toujours inutiles (Paywalled and in French)

Related blog posts

Clearing out the vault by posting preprints

09 December 2024

Bar graphs, how do they work?

I make a brief cameo appearance in a new article about data visualization. Bar graphs are about as simple and basic as you get in data visualization, but a couple of new preprints (one of which is mine) show that people struggle to get even those right. The major preprint by Lin and Landry finds all sorts of issues in bar graphs. Mine is much smaller, and I just want people to label their error bars.

By the way, this was the unexpected call I got after posting a preprint last month.

Related posts

Clearing out the vault by posting preprints

Reference

Heidt A. 2024. Bad bar charts are pervasive across biology. Nature. https://doi.org/10.1038/d41586-024-03996-w

04 December 2024

“Pay me now or pay me later” in reproducibility

“Reproducibility debt” is an interesting and useful take on the matter of reproducibility and replication. I stumbled across a discussion on a recent podcast.

What I like about this phrasing is that a lot of discussion around reproducibility focuses on bad practices. Things like p-hacking, HARKing, and the like. Framing issues around reproducibility as debt makes it more obvious that what we are talking about are trade-offs.

You might have a more reproducible result if you had a bigger sample size and wrote a perfect paper. But that takes time (opportunity costs), and often takes money (financial costs). And there are benefits to getting papers out - both personal (another thing to add to your annual evaluation) and to the community (puts new ideas out and generates leads for others).

In the short term, it can make sense to take on debt. But you will have to pay it back later.

The paper develops a preliminary list of the kinds of trade-offs that cause reproducibility debt. 

  • Data-centric issues (e.g., data storage)
  • Code-centric issues (e.g., code development)
  • Documentation issues (i.e., incomplete or unclear documentation)
  • Tools-centric issues (e.g., software infrastructure)
  • Versioning issues (e.g., code unavailable)
  • Human-centric issues (e.g., lack of funding)
  • Legal issues (e.g., intellectual property conflicts)

It’s very software focused, so I don’t think the list is comprehensive. For example, in biology, reproducibility might become an issue because a species becomes rare or extinct.

If we have reproducibility debt, maybe we can also conceive of reproducibility bankruptcy: a point where the accumulated shortcuts add up to a complete inability to move forward on knowledge.

References

Hassan Z, Treude C, Norrish M, Williams G, Potanin A. 2024. Characterising reproducibility debt in scientific software: A systematic literature review. http://dx.doi.org/10.2139/ssrn.4801433 

Hassan Z, Treude C, Norrish M, Williams G, Potanin A. 2024. Reproducibility debt: Challenges and future pathways. Companion Proceedings of the 32nd ACM International Conference on the Foundations of Software Engineering: 462-466. https://doi.org/10.1145/3663529.3663778

External links

To Be Reproducible or Not To Be Reproducible? That is so Not the Question

29 November 2024

A journal paying for reviews

The BMJ will start paying £50 for reviews.

The catch? It’s not for peer review, it’s for reviews from patients and the public. They still expect peer reviews to be done as part of service.

Their announcement post does not describe how they will address several of what I consider to be potential problems. While my previous post was about posting for reviews from academics, I think many of the issues will also apply to public and patient review.

I’m going to sign up as a public reviewer. Because why not? I could use 50 quid. And I’d like to see what this looks like from the inside.

Related posts

Potential problems of paying peer reviewers

External links

The BMJ will remunerate patient and public reviewers