29 November 2024

A journal paying for reviews

The BMJ will start paying £50 for reviews.

The catch? It’s not for peer review, it’s for reviews from patients and the public. They still expect peer reviews to be done as part of service.

Their announcement post does not describe how they will address several of what I consider to be potential problems. While my previous post was about posting for reviews from academics, I think many of the issues will also apply to public and patient review.

I’m going to sign up as a public reviewer. Because why not? I could use 50 quid. And I’d like to see what this looks like from the inside.

Related posts

Potential problems of paying peer reviewers

External links

The BMJ will remunerate patient and public reviewers

25 November 2024

“Neurosurgery on Saturn” paper shows academics’ blind spots

The planet Saturn.
In the last few days, a bunch of people on Bluesky discovered a paper that has been out for a few months, “Practice of neurosurgery on Saturn.”

Looking through the social media posts about this paper, a lot of people played that favourite academic game, “How did this get published?” Many people suggested it’s a hoax. Academic hoaxes are a particular interest of mine, and I am always looking for the next entry in the Stinging the Predators collection of academic hoaxes.

I didn’t think this was a hoax? Hoaxers usually reveal the prank almost immediately, and this paper had been out for months.

My hunch seems to have been correct. The lead author has an extensive Google Scholar page and said on PubPeer: “it is clear that the document focuses on fictitious and hypothetical situations.” I am not clear about what the point of the article is, but never mind. It’s filed under “Letters to the editor,” which I think is an arena where researchers and journals can be allowed a little leeway.

But this is a good example of something that I think is decidedly lacking in many discussions about academic publishing and academic integrity. In none of the posts I read did anyone do any actual investigation.

Nobody looked to see if the authors were real.

Nobody emailed the authors. It seems to just be happenstance that the lead author saw the PubPeer comments and replied.

Nobody emailed the journal (although editors are often notoriously slow to reply to these sorts of things).

In collecting academic hoaxes, I’ve noticed a similar pattern. People create hoaxes to show that there are bad journals out there that accept anything for money. But by and large, that is where it stops

People know predatory journals are out there, but nobody is actively digging behind the scenes to see how they work. How do people decide to start running them? How much money do they make? Why would a scam artist only in it for the money do apparently counterintuitive things like waiving the article processing charges? (There are multiple instances of that in the Stinging the Predators collection.)

A recent paper came out that made a similar point about the lack of investigation about paper mills  (Byrne et al. 2024). 

Academics treat too many of these problems around dubious publishing as some sort of black box that cannot be opened. They only study the outputs. I think someone needs to approach these sorts of problems more like an investigative journalist or an undercover law enforcement officer might.

Go in and find the people who are doing all this dishonest stuff. Get them talking. I want hear what some of the people organizing predatory journals or paper mills or citations rings have to say. Why do they do what they do?

I don’t expect academics themselves to do this. This kind of investigative journalism is expensive and time consuming and being done less and less. But without this kind of insight, we will probably never be able to understand and curb these problems.

References

Byrne JA, Abalkina A, Akinduro-Aje O, Christopher J, Eaton SE, Joshi N, Scheffler U, Wise NH, Wright J. 2024. A call for research to address the threat of paper mills. PLoS Biology 22(11): e3002931. https://doi.org/10.1371/journal.pbio.3002931

Mostofi K, Peyravi M. 2024. Practice of neurosurgery on Saturn. International Journal of Surgery Case Reports 122: 110139. https://doi.org/10.1016/j.ijscr.2024.110139

External links

PubPeer commentary on “Practice of neurosurgery on Saturn”

Altmetric page for “Practice of neurosugery on Saturn”

Google Scholar page for Keyvan Mostofi

Photo by Steve Hill on Flickr, used under a Creative Commons licence. 

Generative AI: Two of of three is bad

There’s an old joke: “Fast, cheap, good: Pick two.”

Generative AI is fast and cheap. It will not be good.

I would like “good” to be a higher priority.

Edit, 1 December 2024: I think academics, under the intense pressure to be productive, prioritize those three characteristics in that order. The first two might switch (grad students are more likely to prioritize “cheap”), but I think “good” is regularly coming last.



22 November 2024

Clearing out the vault by posting preprints

Old books on a shelf
My last jobs didn’t have any expectations for research, so my publication rate slowed significantly. I wanted to do something about that.

A few months ago, I published a submitted but not completed article on paying peer reviewers here on the blog and on Figshare. This got me digging around in folders on my hard drive and I got thinking about other articles that I had written, submitted, but weren’t accepted. And at the time, I just ran out of steam to revise and resubmit them in ways that would satisfy an editor and a couple of reviewers. 

So in recent weeks, I have taken to submitting some of those manuscripts to a preprint server.

In the past, I was a little cool to the value of preprints. I always thought they had some uses, but I was skeptical that they would replace journals, which was the stated wish for some of the strongest preprint advocates. I was always worried about the Matthew effect: that famous labs would benefit from preprints and everyone else would just have more work to do.

But since I last wrote about preprints, the attention landscape has changed. More people in biology have started scanning through preprint servers part of their routine. I was surprised when a reporter emailed me about one of these preprints and wanted to chat for an interview. I don’t think that would have happened a few years ago.

Whether these will be “final” public version of these little projects, I can’t say. I have other projects that I want to get out that have not been written up at all yet that I want to try to get into a journal. But I am glad that every one of these received at least a little attention on Bluesky.

Here’s the list of my recent preprints, all on bioRxiv.

Update, 26 November 2024: I’ve now had two journalists reach out to me because of these preprints. I’ve never had that high a level of interest from the dozens of peer-reviewed journal articles I published.

Related posts

A pre-print experiment: will anyone notice?

A pre-print experiment, continued  

A pre-print experiment, part 3: Someone did notice  

Photo by umjadedoan on Flickr. Used under a Creative Commons license.

18 November 2024

The cat that fooled Google Scholar, the newest hoax in my collection

I finally got a chance to update my collection of academic hoaxes!

Stinging the Predators 23.0: Now with a cat! (The Internet loves cats, right?) http://bit.ly/StingPred

I am now up to 42 academic hoaxes, which is triple the number that I started with in version 1.0 in 2017.

The latest hoax targets not a predatory journal, but an academic search engine. While this is unusual, it is not the first hoax that was pulled to show how easy it is to manipulate Google Scholar. (One of the things that has been interesting to me as this project has continued is that many hoaxers feel compelled to make the same point again.)

And one other things that has been rewarding is that this collection, which I’ve only ever had on Figshare and promoted on my socials and personal websites, has been viewed tens of thousands of times and has been cited by scholars writing in proper journals a few time.

External links

Faulkes Z. 2024. Stinging the Predators: A Collection of Papers That Should Never Have Been Published (version 23). Figshare. https://doi.org/10.6084/m9.figshare.5248264



14 November 2024

Okay, stop. Saying “science isn’t political” will not keep science safe from political attacks

Advice from people with experience fighting powerful fascist opponents: “Do not comply in advance.

In a new Science editorial, National Academies of Sciences, Engineering, and Medicine president Marcia McNutt starts complying well in advance. 

The editorial begins with blatant concern trolling:

I had become ever more concerned that science has fallen victim to the same political divisiveness tearing at the seams of American society.

Okay, stop. McNutt provides zero examples of supposed “divisiveness” of the scientific community. A recent Nature article showed exactly the opposite. The scientific community was strongly united in its preferred American election outcome: 86% favoured the losing candidate. I guess McNutt sees that as bad because it makes it harder for her to play nice with with administration of the winning candidate.

(The scientific community) must take a critical look at what responsibility it bears in science becoming politically contentious(.)

Okay, stop. Again, McNutt provides zero examples of scientists making science politically contentious. On the other hand, I can point to many examples of politicians who waded into scientific debates about the reality of whole branches of science (evolution, geology, climate science) and health care.

No, the problem (according to McNutt) is that we scientists don’t explain ourselves well. We don’t tell politicians and citizens about how we are just disinterested third parties.

(S)cience, at its most basic, is apolitical.

Okay, stop. I know this is a popular claim, but it’s time to put this into the ground and bury it six feet deep. There is a very good Nature podcast series that takes this claim apart. Claiming “We’re not political” is a fiction that favours those who are politically privileged. McNutt would have had a much stronger case if she had said that science is not partisan. Or that reality is apolitical.

But science is an organized profession done by humans, so science is political.

It is strange to say that science is apolitical when I see the two entangled all the time.

McNutt continues:

Whether conservative or liberal, citizens ignore the nature of reality at their peril.

Okay, stop. Many elected politicians and citizens are just fine with ignoring the nature of reality – as long as they personally are not affected. And those who are personally affected, in genuine peril, may not be able to generate enough political clout to change policy to make themselves safe on their own. They need allies.

McNutt’s argument that “the arc of the scientific universe is long, but it bends towards truth” is callous. It seems like her view is that scientists should passively sit on the sidelines, presenting data but never advocating, while watch people make the same mistakes about discredited claims that actively harm people over and over.

I can’t help but wonder where McNutt has been for the last decade or so when she can, apparently in all seriousness, write a sentence like this about climate science: 

It is up to society and its elected leadership to decide how to balance these options, including the use of renewable energy, climate adaptation, carbon capture, or even various interventions that reflect sunlight back into space.

Okay, stop.

Does McNutt understand that the incoming elected leadership repeatedly stated that their option is, “Climate science is all a hoax. We don’t need to do anything”?

Does McNutt understand that elected leaders can use their power to take actions that are not supported by the majority of society or scientific evidence?

Should scientists simply accept that their elected leadership is condemning millions to ever greater misery every day this denial of reality goes on?

McNutt is just trying to get her organization out of the line of fire of an incoming government that is more overtly hostile to science than maybe any other American government ever. 

The NAS stands ready, as it always has, to advise the incoming administration.

While it is the job of a civil servant to work with a new boss after every election, most scientists are not civil servants. They are not obligated to support a new government. They can, and should, do much more than just provide data and hope that elected leaders eventually come around to face scientific reality. If anyone is not coming around to face reality – the political reality, in this case – it’s McNutt.

External links

Science is neither red nor blue

The US election is monumental for science, say Nature readers — here’s why

“Stick to the science”: When science gets political (Three part podcast series)

06 November 2024

Untitled post 2024

You get hurt, hurt ‘em back. You get killed…

Walk it off.

 Captain America, Avengers: Age of Ultron

05 November 2024

Gen AI fatalism

Generative AI fatalism (or Gen AI fatalism): The assertion that generative AI must be accepted, because its widespread adoption is inevitable and cannot be stopped.

A new article by James Zuo about ChatGPT in peer review is a particularly spectacular example of gen AI fatalism. Zuo mentions that many peer reviews show signs of being created by large language models, like ChatGPT. He lists many problems and only trivial potential advantages, and calls for more human interactions in the peer review process.

Since Zuo has nothing very positive to say about generative AI in peer review, the fatalism on display is stark.

The tidal wave of LLM use in academic writing and peer review cannot be stopped.

No! 

This is a mere assertion that has no basis behind it. Technology adoption and use is not some sort of physical law. It’s not a constant like the speed of light. It’s not the expansion of the universe driven by the Big Bang, dark matter, and dark energy.

The adoption and use of technology is a choice. Technologies fail in the marketplace all the time. We abandon technologies all the time.

If generative AI is causing problems in the peer review process, we should say that. We should work to educate our colleagues about the inherent problems with generative AI in the review process.

I suspect that people use ChatGPT for the simple reason that they don’t want to do peer review. They do not believe they are adequately rewarded for the task. So, as is so often the case, we need to look at academic incentives. We need to look at why we peer review and what it can accomplish.

Creating journal policies about using ChatGPT is little more than trying to paper up holes in a boat. I would welcome the equivalent of pulling the boat into dry dock and doing a complete refit.

Reference

https://doi.org/10.1038/d41586-024-03588-8

Stay out of my academic searches, gen AI

Something I had long dreaded came to pass yesterday.


Google Scholar landing page with " New! AI outlines in Scholar PDF Reader: skim the bullets, deep read what you need"
Google Scholar introduced generative AI.

“New! AI outlines in Scholar PDF Reader: skim the bullets, deep read what you need.”

Andrew Thaler had the perfect riposte to this new feature.

If only scholarly publications came with a short synopsis of the paper right up front, written by the authors to highlight the important and salient points of the study.

We could give it a nifty name, like “abstract.”
Exactly! Not only do researchers already outline papers, many journals require two such outlines: an abstract and some sort of plain English summary.

I don’t need this. I don’t want this. No more generative AI infiltrating into every nook and cranny of the Web, please.

Punching fascists: A part of our heritage

Apropos of nothing, here are two pictures I thought I’d posted back in November 2016, but I can’t seem to find any more.

They’re a couple of panels of Canadian World War II comic character Johnny Canuck punching Hitler.

Johnny Canuck punching Hitler.

Johnny Canuck punching Hitler.