A journal paying for reviews

The BMJ will start paying £50 for reviews.

The catch? It’s not for peer review, it’s for reviews from patients and the public. They still expect peer reviews to be done as part of service.

Their announcement post does not describe how they will address several of what I consider to be potential problems. While my previous post was about posting for reviews from academics, I think many of the issues will also apply to public and patient review.

I’m going to sign up as a public reviewer. Because why not? I could use 50 quid. And I’d like to see what this looks like from the inside.

Related posts

Potential problems of paying peer reviewers

External links

The BMJ will remunerate patient and public reviewers

“Neurosurgery on Saturn” paper shows academics’ blind spots

In the last few days, a bunch of people on Bluesky discovered a paper that has been out for a few months, “Practice of neurosurgery on Saturn.”

Looking through the social media posts about this paper, a lot of people played that favourite academic game, “How did this get published?” Many people suggested it’s a hoax. Academic hoaxes are a particular interest of mine, and I am always looking for the next entry in the Stinging the Predators collection of academic hoaxes.

I didn’t think this was a hoax? Hoaxers usually reveal the prank almost immediately, and this paper had been out for months.

My hunch seems to have been correct. The lead author has an extensive Google Scholar page and said on PubPeer: “it is clear that the document focuses on fictitious and hypothetical situations.” I am not clear about what the point of the article is, but never mind. It’s filed under “Letters to the editor,” which I think is an arena where researchers and journals can be allowed a little leeway.

But this is a good example of something that I think is decidedly lacking in many discussions about academic publishing and academic integrity. In none of the posts I read did anyone do any actual investigation.

Nobody looked to see if the authors were real.

Nobody emailed the authors. It seems to just be happenstance that the lead author saw the PubPeer comments and replied.

Nobody emailed the journal (although editors are often notoriously slow to reply to these sorts of things).

In collecting academic hoaxes, I’ve noticed a similar pattern. People create hoaxes to show that there are bad journals out there that accept anything for money. But by and large, that is where it stops. 

People know predatory journals are out there, but nobody is actively digging behind the scenes to see how they work. How do people decide to start running them? How much money do they make? Why would a scam artist only in it for the money do apparently counterintuitive things like waiving the article processing charges? (There are multiple instances of that in the Stinging the Predators collection.)

A recent paper came out that made a similar point about the lack of investigation about paper mills  (Byrne et al. 2024). 

Academics treat too many of these problems around dubious publishing as some sort of black box that cannot be opened. They only study the outputs. I think someone needs to approach these sorts of problems more like an investigative journalist or an undercover law enforcement officer might.

Go in and find the people who are doing all this dishonest stuff. Get them talking. I want hear what some of the people organizing predatory journals or paper mills or citations rings have to say. Why do they do what they do?

I don’t expect academics themselves to do this. This kind of investigative journalism is expensive and time consuming and being done less and less. But without this kind of insight, we will probably never be able to understand and curb these problems.

References

Byrne JA, Abalkina A, Akinduro-Aje O, Christopher J, Eaton SE, Joshi N, Scheffler U, Wise NH, Wright J. 2024. A call for research to address the threat of paper mills. PLoS Biology 22(11): e3002931. https://doi.org/10.1371/journal.pbio.3002931

Mostofi K, Peyravi M. 2024. Practice of neurosurgery on Saturn. International Journal of Surgery Case Reports 122: 110139. https://doi.org/10.1016/j.ijscr.2024.110139

External links

PubPeer commentary on “Practice of neurosurgery on Saturn”

Altmetric page for “Practice of neurosugery on Saturn”

Google Scholar page for Keyvan Mostofi

Photo by Steve Hill on Flickr, used under a Creative Commons licence. 

Generative AI: Two of of three is bad

There’s an old joke: “Fast, cheap, good: Pick two.”

Generative AI is fast and cheap. It will not be good.

I would like “good” to be a higher priority.

Edit, 1 December 2024: I think academics, under the intense pressure to be productive, prioritize those three characteristics in that order. The first two might switch (grad students are more likely to prioritize “cheap”), but I think “good” is regularly coming last.

Clearing out the vault by posting preprints

My last jobs didn’t have any expectations for research, so my publication rate slowed significantly. I wanted to do something about that.

A few months ago, I published a submitted but not completed article on paying peer reviewers here on the blog and on Figshare. This got me digging around in folders on my hard drive and I got thinking about other articles that I had written, submitted, but weren’t accepted. And at the time, I just ran out of steam to revise and resubmit them in ways that would satisfy an editor and a couple of reviewers. 

So in recent weeks, I have taken to submitting some of those manuscripts to a preprint server.

In the past, I was a little cool to the value of preprints. I always thought they had some uses, but I was skeptical that they would replace journals, which was the stated wish for some of the strongest preprint advocates. I was always worried about the Matthew effect: that famous labs would benefit from preprints and everyone else would just have more work to do.

But since I last wrote about preprints, the attention landscape has changed. More people in biology have started scanning through preprint servers part of their routine. I was surprised when a reporter emailed me about one of these preprints and wanted to chat for an interview. I don’t think that would have happened a few years ago.

Whether these will be “final” public version of these little projects, I can’t say. I have other projects that I want to get out that have not been written up at all yet that I want to try to get into a journal. But I am glad that every one of these received at least a little attention on Bluesky.

Here’s the list of my recent preprints, all on bioRxiv.

• Trends in hoaxes of academic communication. 🔊 Audio summary
• Uninterpretable graphs at a scientific conference. 🔊 Audio summary 
• When crayfish make news, headlines are correct but still misleading. 🔊 Audio summary:
  

Update, 26 November 2024: I’ve now had two journalists reach out to me because of these preprints. I’ve never had that high a level of interest from the dozens of peer-reviewed journal articles I published.

Related posts

A pre-print experiment: will anyone notice?

A pre-print experiment, continued  

A pre-print experiment, part 3: Someone did notice  

Photo by umjadedoan on Flickr. Used under a Creative Commons license.

The cat that fooled Google Scholar, the newest hoax in my collection

I finally got a chance to update my collection of academic hoaxes!

I am now up to 42 academic hoaxes, which is triple the number that I started with in version 1.0 in 2017.

The latest hoax targets not a predatory journal, but an academic search engine. While this is unusual, it is not the first hoax that was pulled to show how easy it is to manipulate Google Scholar. (One of the things that has been interesting to me as this project has continued is that many hoaxers feel compelled to make the same point again.)

And one other things that has been rewarding is that this collection, which I’ve only ever had on Figshare and promoted on my socials and personal websites, has been viewed tens of thousands of times and has been cited by scholars writing in proper journals a few time.

External links

Faulkes Z. 2024. Stinging the Predators: A Collection of Papers That Should Never Have Been Published (version 23). Figshare. https://doi.org/10.6084/m9.figshare.5248264

Okay, stop. Saying “science isn’t political” will not keep science safe from political attacks

Advice from people with experience fighting powerful fascist opponents: “Do not comply in advance.”

In a new Science editorial, National Academies of Sciences, Engineering, and Medicine president Marcia McNutt starts complying well in advance. 

The editorial begins with blatant concern trolling:

I had become ever more concerned that science has fallen victim to the same political divisiveness tearing at the seams of American society.

Okay, stop. McNutt provides zero examples of supposed “divisiveness” of the scientific community. A recent Nature article showed exactly the opposite. The scientific community was strongly united in its preferred American election outcome: 86% favoured the losing candidate. I guess McNutt sees that as bad because it makes it harder for her to play nice with with administration of the winning candidate.

(The scientific community) must take a critical look at what responsibility it bears in science becoming politically contentious(.)

Okay, stop. Again, McNutt provides zero examples of scientists making science politically contentious. On the other hand, I can point to many examples of politicians who waded into scientific debates about the reality of whole branches of science (evolution, geology, climate science) and health care.

No, the problem (according to McNutt) is that we scientists don’t explain ourselves well. We don’t tell politicians and citizens about how we are just disinterested third parties.

(S)cience, at its most basic, is apolitical.

Okay, stop. I know this is a popular claim, but it’s time to put this into the ground and bury it six feet deep. There is a very good Nature podcast series that takes this claim apart. Claiming “We’re not political” is a fiction that favours those who are politically privileged. McNutt would have had a much stronger case if she had said that science is not partisan. Or that reality is apolitical.

But science is an organized profession done by humans, so science is political.

It is strange to say that science is apolitical when I see the two entangled all the time.

McNutt continues:

Whether conservative or liberal, citizens ignore the nature of reality at their peril.

Okay, stop. Many elected politicians and citizens are just fine with ignoring the nature of reality – as long as they personally are not affected. And those who are personally affected, in genuine peril, may not be able to generate enough political clout to change policy to make themselves safe on their own. They need allies.

McNutt’s argument that “the arc of the scientific universe is long, but it bends towards truth” is callous. It seems like her view is that scientists should passively sit on the sidelines, presenting data but never advocating, while watch people make the same mistakes about discredited claims that actively harm people over and over.

I can’t help but wonder where McNutt has been for the last decade or so when she can, apparently in all seriousness, write a sentence like this about climate science: 

It is up to society and its elected leadership to decide how to balance these options, including the use of renewable energy, climate adaptation, carbon capture, or even various interventions that reflect sunlight back into space.

Okay, stop.

Does McNutt understand that the incoming elected leadership repeatedly stated that their option is, “Climate science is all a hoax. We don’t need to do anything”?

Does McNutt understand that elected leaders can use their power to take actions that are not supported by the majority of society or scientific evidence?

Should scientists simply accept that their elected leadership is condemning millions to ever greater misery every day this denial of reality goes on?

McNutt is just trying to get her organization out of the line of fire of an incoming government that is more overtly hostile to science than maybe any other American government ever. 

The NAS stands ready, as it always has, to advise the incoming administration.

While it is the job of a civil servant to work with a new boss after every election, most scientists are not civil servants. They are not obligated to support a new government. They can, and should, do much more than just provide data and hope that elected leaders eventually come around to face scientific reality. If anyone is not coming around to face reality – the political reality, in this case – it’s McNutt.

External links

Science is neither red nor blue

The US election is monumental for science, say Nature readers — here’s why

“Stick to the science”: When science gets political (Three part podcast series)

Untitled post 2024

You get hurt, hurt ‘em back. You get killed…

Walk it off.

 Captain America, Avengers: Age of Ultron

Gen AI fatalism

Generative AI fatalism (or Gen AI fatalism): The assertion that generative AI must be accepted, because its widespread adoption is inevitable and cannot be stopped.

A new article by James Zuo about ChatGPT in peer review is a particularly spectacular example of gen AI fatalism. Zuo mentions that many peer reviews show signs of being created by large language models, like ChatGPT. He lists many problems and only trivial potential advantages, and calls for more human interactions in the peer review process.

Since Zuo has nothing very positive to say about generative AI in peer review, the fatalism on display is stark.

The tidal wave of LLM use in academic writing and peer review cannot be stopped.

No! 

This is a mere assertion that has no basis behind it. Technology adoption and use is not some sort of physical law. It’s not a constant like the speed of light. It’s not the expansion of the universe driven by the Big Bang, dark matter, and dark energy.

The adoption and use of technology is a choice. Technologies fail in the marketplace all the time. We abandon technologies all the time.

If generative AI is causing problems in the peer review process, we should say that. We should work to educate our colleagues about the inherent problems with generative AI in the review process.

I suspect that people use ChatGPT for the simple reason that they don’t want to do peer review. They do not believe they are adequately rewarded for the task. So, as is so often the case, we need to look at academic incentives. We need to look at why we peer review and what it can accomplish.

Creating journal policies about using ChatGPT is little more than trying to paper up holes in a boat. I would welcome the equivalent of pulling the boat into dry dock and doing a complete refit.

Reference

https://doi.org/10.1038/d41586-024-03588-8

Stay out of my academic searches, gen AI

Something I had long dreaded came to pass yesterday.

Google Scholar introduced generative AI.

“New! AI outlines in Scholar PDF Reader: skim the bullets, deep read what you need.”

Andrew Thaler had the perfect riposte to this new feature.

If only scholarly publications came with a short synopsis of the paper right up front, written by the authors to highlight the important and salient points of the study.

We could give it a nifty name, like “abstract.”

Exactly! Not only do researchers already outline papers, many journals require two such outlines: an abstract and some sort of plain English summary.

I don’t need this. I don’t want this. No more generative AI infiltrating into every nook and cranny of the Web, please.

Punching fascists: A part of our heritage

Apropos of nothing, here are two pictures I thought I’d posted back in November 2016, but I can’t seem to find any more.

They’re a couple of panels of Canadian World War II comic character Johnny Canuck punching Hitler.

Research on crustacean nociceptors missing useful evidence

A new paper by Kasiouras and colleagues attempts to provide evidence for nociceptive neurons in a decapod crustacean, a shore crab. 

The authors took crabs, poked them and dropped some acid on them, and recorded the neural responses. Neurons did indeed respond, in dose-and intensity specific ways. From this, the authors conclude these are potential nociceptive responses.

I am unconvinced. There is a big difference between showing that neurons can respond to a possibly noxious stimulus and showing that those neurons are responding as nociceptors.

First:

The same category of stimuli can have detected by by nociceptors and by “regular” sensory neurons that are not nociceptors. For example, an increase in temperature can stimulate both nociceptors and thermoreceptive sensory neurons. Mechanical pressure can stimulate both mechanoreceptors and nociceptors. That there are two neural pathways are probably why we distinguish a burn (painful) as different than heat, or a pinch from a strong touch.

The results would be more convincing if the authors showed that neurons responded in ways that are typical of other nociceptors. Many (though not all) nociceptors have a few common properties.

1. They respond to several different types of stimuli. For example, they react to high temperature and acid and mechanical pressure and a chemical like capsaicin. (The technical term is that they are polymodal or  multimodal sensory neurons.)
   
2. They respond to repeated stimulation with increased activity. Most sensory neurons “get used to” the same sensory stimuli over time, but many nociceptors do the opposite. (The technical term is that they show sensitization.)

The authors couldn’t do either of these, because they are recording from the whole nerve cord and they did not pick out the activity of single neurons. Sometimes it is possible to recognize the activity of single neurons in this sort of record with spike sorting techniques, but that is done in this paper.

Second:

Species respond to potentially noxious stimuli in different ways. Some species respond to capsaicin with nociceptive behaviours, but others do not. No stimulus is guaranteed trigger nociceptors in all cases. 

This paper used mechanical touch and acetic acid, but because the paper did no behavioural experiments, it’s not clear if the crabs perform nociceptive behaviour in response to the level of stimuli presented.

Another paper used acetic acid as a potentially noxious stimulus with shore crabs, and crabs do respond to it (Elwood et al. 2017), but that paper was criticized (Diggles 2018; not cited in the current study) for not considering the possibility that acetic acid caused a feeding (gustatory) response, not a nociceptive response. Acetic acid is the technical name for vinegar, after all.

The results would be more convincing if the electophysiological recordings of neurons were connected back to the crab’s behaviour. For example, the authors could tried an experiment to shown that a touch of low intensity caused one sort of behaviour, but a touch of higher intensity caused a different behaviour that looks like nociceptive behaviour. Then they could have seen what the differences in neural activity to those two kinds of tactile stimulation were.

I am glad to see more labs trying to establish the presence or absence of nociceptors in crustaceans. There is still more work to demonstrate their existence and characterize their physiology if they exist.

References

Diggles BK. 2018. Review of some scientific issues related to crustacean welfare. ICES Journal of Marine Science: fsy058. http://dx.doi.org/10.1093/icesjms/fsy058
 

Elwood RW, Dalton N, Riddell G. 2017. Aversive responses by shore crabs to acetic acid but not to capsaicin. Behavioural Processes 140: 1-5. https://doi.org/10.1016/j.beproc.2017.03.022

Kasiouras E, Hubbard PC, Gräns A, Sneddon LU. 2024. Putative nociceptive responses in a decapod crustacean: The shore crab (Carcinus maenas). Biology 13(11): 851. https://doi.org/10.3390/biology13110851
 

Picture by Dunnock_D on Flickr; used under a Creative Commons license.

Pursing integrity over excellence in research assessment

I was reading yet another “Scientists behaving badly” article. This one was about Jonathan Pruitt, who used to work where I used to work (different departments). And, as it usual in these articles, there is a section about how institutions assess research:

Many attribute the rising rate of retractions to academia’s high-pressure publish-or-perish norms, where the only route to a coveted and increasingly scarce permanent job, along with the protections of tenure and better pay, is to produce as many journal articles as possible. That research output isn’t just about publishing papers but also how often they’re cited. Some researchers might try to bring up their h-index (which measures scientific impact, based on how many times one is published and cited), or a journal might hope that sharing particularly enticing results will enhance its impact factor (calculated using the number of citations a journal receives and the number of articles it publishes within a two-year period).

It finally occurred to me that focusing on the indicators like citations and Impact Factor are all symptoms of a larger mindset.

The major watchword for administrators and funders for decades has been “excellence.” Some prefer a near synonym, “impact.” Everyone wanted to reward excellent research, which is an easy sell. Nobody wants to admit that they want to support average research — even though that’s what most research is, by definition. But most of science progresses by millimetres on the basis of average research. Even poor research can have some nuggets of data or ideas that can be useful to others.

I suggest a new overarching principle to guide assessment: integrity. We should be paying attention to, and rewarding research and researchers that act in the most trustworthy ways. Who are building the most solid and dependable research. That can be assessed by practices like data sharing and code sharing, and by showing of community use and replication.

The pursuit of excellence has proved too fickle and too likely to make people act sketchy to become famous. Let’s change the target.

Update, 28 November 2024: Blog post by Grace Gottleib on how to assess integrity.

External links

A rock-star researcher spun a web of lies—and nearly got away with it