25 March 2025

A second test of paid peer review

After years of sloganeering, it’s interesting to see two trials of paid peer review coming on the heels of one another. The latest test of paying peer reviewers was run by the journal Biology Open (where I have published) using about 20 manuscripts. 

Reviewers got paid somewhere around US$259-285 for doing a peer review that had to come back promptly. As in, within a week. And they did, with no apparently lapse in quality.

This looks promising, but some issues that are not caught in a small trial could emerge if payment becomes institutionalized.

Got to get back to prepping for class.

Update, 26 March 2025: The pseudonymous Overly Honest Editor on Bluesky writes (light edits):

I’m seeing £44k for 20 papers after discounting pay for refs. That is £2200 per paper, not far off from many mid-range journals APCs. Which to me translates to the usual: yes, we could pay peer reviewers, if people are willing to cash out 30% extra. And people aren’t willing.

Reference

Gorelick DA, Clark A. 2025. Fast & Fair peer review: a pilot study demonstrating feasibility of rapid, high-quality peer review in a biology journal. bioRxiv: 2025.03.18.644032.

Related posts

A test of pay for peer review 

Potential problems of paying peer reviewers

17 March 2025

A test of pay for peer review

“Pay reviewers” has become a social media slogan that could be a welcome change to academic publishing if potential pitfalls are avoided. But because we are researchers, we need to do the experiment. A recent paper by Cotton and colleagues (2025) is the first I have seen to test what paying peer reviewers might accomplish. (There was a prior test by Chetty and colleagues in 2014.)

The journal Critical Care Medicine offered reviewers an incentive of $250 on alternate weeks.

Does offering to pay peer reviewers get more reviews? The authors measure this in three ways.

  1. Percent of invitations accepted. There was no difference between offering payment or not.
  2. “Rate of conversion,” that is, the percent of reviews actually submitted. Offering an incentive improved this measure.
  3. “On time rate of conversion,” which was the percent of reviews submitted within two weeks. This is probably the most relevant measure for a journal, since timely review is important. This was again improved by offering an incentive: about 42% of invitations got a review back in two weeks when offered the incentive, compare to about 32% without. The average difference in time to return a review is a day.

By the way, for those 32% of reviewers who got a review done within two weeks for no pay, I appreciate you. Not all heroes wear capes.

Is peer review quality affected? Not noticeably. The results note that about a third of the reviews that were received did not got a quality assessment. By reading between the lines, it appears that the handling associate editors were only “encouraged” to grade the reviews, and a chunk of the editors just did not rate the reviews.

There is a difference by offering payment, but is it worth it? Trevor Branch noted:

(T)hese results (8% higher rate of completing review, 1 day earlier on average) don't exactly motivate journals to pay reviewers.

Cotton and colleagues notes that the costs add up.

(F)or a journal with 1,000 reviewed submissions per year, three paid reviewers per article, and payments of $250 per reviewer, such a payment mechanism could cost $750,000 annually.

Even if you ramped it down to two reviewers, which is common for many basic science journals, it’s not clear that the cost-benefit ratio of paying peer reviewers is worth it for journals.

This study can’t address many of the other possible problems that will undoubtedly emerge. For example, what will authors say if they found if their article is rejected on the basis of what they think is a shoddy review that the journal paid for? What is an appropriate amount of incentive?

I recognize that for many, paying for peer review is about moral fairness, not financial prudence.

On a related note, Stat News has an update on the lawsuit launched against scientific publishers last year. The publishers are asking for a dismissal for similar reasons that I outlined last year.

Legal experts told STAT that while there’s little question unpaid peer review has helped enrich publishers, it may be difficult to prove the practices cited in the lawsuit were the product of an anticompetitive agreement.

Much of the article, however, is as much about about the service model of peer review as the lawsuit itself. The service model of peer review is under strain, but I don’t think the article articulates the reasons why very well.

First and foremost, the assessment schemes of most research institutions are weighed far too heavily towards output. 

Second, the globalization of science means that the community of peers is more diffuse. Peere review works best, I think, when many people know each other.

Third, there are likely still imbalances in who is asked to review papers (e.g., established researchers from established institutions in countries with colonial histories are asked more often than researchers everywhere else).

References

Chetty R, Saez E, Sándor L. 2014. What policies increase prosocial behavior? An experiment with referees at the journal of public economics. Journal of Economic Perspectives 28:169–188. https://doi.org/10.1257/jep.28.3.169

Cotton CS, Alam A, Tosta S, Buchman TG, Maslove DM. Effect of monetary incentives on peer review acceptance and completion: A quasi-randomized interventional trial. Critical Care Medicine: in press. http://doi.org/10.1097/CCM.0000000000006637

Related posts

Scholarly publishers sued 

Potential problems of paying peer reviewers

18 February 2025

How to shape public opinion

 Shana Gadarian just wrote this useful thread on Bluesky. I’ve compiled it with some light editing.

Some brief thoughts from someone who studies public opinion.

The federal firing spree can be made unpopular because it is being done in a haphazard way, there will be huge economic effects not just in DC but also across country and it’s also likely illegal in many cases. But how to make that argument?

First, remember that most people pay attention to politics most of the time and don’t really know how things work so first explain to people how the federal government works. Really! Who allocates money, who gets to spend it, and why is it a problem for an elected person to be firing civil servants.

Much of our policymaking is done in basically invisible ways so it’s not always clear to people what the policies are that affect their lives. (See The Submerged State by Suzanne Mettler.)

Tell people what current policy is and how long it’s been there. People want policy information but if they don't have it, they fill in the details. (See The Invented State by Emily Thorson.) Right now, the current regime is telling them that government is wasteful and inefficient.

Focus on both the immediate effects of federal funding cuts on the health and welfare of specific people and the country as a whole. Tell both sets of stories! What happens to federal workers at NIH when they lose their jobs, their research, and what are the effects on cancer patients.

It is really hard to track things that don’t happen, but also think about and articulate scenarios of what might happen if PepFar doesn’t fund HIV treatments. Babies develop AIDS. Other countries, like China, fill in the gaps of where the US was and win influence in these regions.

Focus on the very unpopular aspects of the cuts. You can’t go to national parks if there are no park rangers and it’s unsafe to go there. National parks are very popular!

People want things to work and they want to be safe. Talk about the direct consequences of things like cutting FAA staff for airline safety. It would be helpful for pilots, flight attendant, and crew unions to be doing this publicly and also privately through lobbying.

Opinion works when it’s collective, so helping people to understand their own values map on to what is happening and how to then translate that into political action. Calls and emails to their members, working with membership groups, letters to the editor, protest. There are many ways to make voices count.

Trace out the economic effects in very local ways. How many people in your local area work for federal agencies. Will farmers stop getting subsidies? Will disabled students stop getting needed services? Which hospitals will have to stop clinical trials? Which people are affected the most?

 


The missing science solidarity

So. In the United States, funding for science has slowed, mass layoffs have cut through federal science agencies like a chainsaw, and someone who doesn’t believe that germs cause disease is leading the country’s health agency.

Are we scientists reacting with anywhere near enough unity to fight back?

Christina Pagel wrote (lightly edited):

Another group who have done little to organise an opposition are scientists and scientific societies.

While there are ongoing lawsuits around pauses in grant funding or pausing international health programmes, these are being driven by labour organizations or non-governmental organizations, not scientists. Instead, some journals, organizations, universities are quietly censoring output of targeted terms like “gender.” Publicly funded bodies such as NASA, NIH and NSF have little choice but to comply. Individual scientists have spoken out, but many are also reporting self-censoring in fear of impact on their careers. 

Possible reasons:

  1. The scientific community is spread across universities, private research institutes, and government agencies and across very different disciplines, making coordinated action – or even agreeing on aligned interests – challenging.
  2. Large national grant making bodies are not independent of the federal government. Scientists often require a funder’s permission to publish and have little formal say in priority areas for funding.
  3. Most scientists just want to work and aren’t “into politics.” Most fields are relatively unaffected.
  4. Many scientists also feel an instinctive aversion to advocacy, worried that it is not scientific or objective enough.
  5. Some who benefit from changes in funding priorities will welcome the opportunity to further their own research.

We saw all these factors play out in the ideological bent to science in Nazi Germany or Stalinist Russia.

The stability of recent decades has left scientists unprepared for when their independence is revealed to be so fragile with few if any legal protections.

Pagels’s list is helpful start, to which I will add.

Point 1: That the scientific community is spread across institutions is one of the reasons we have scientific societies. Academic societies can represent thousands, or tens of thousands, of researchers in the United States and around the world. But the response of scientific societies to the catastrophic events of the last few weeks has been – at best – mixed. 

Scientific societies have the potential to be critical advocates for science in the United States, and organize collective action – by which I mean protests – against the budget cuts and firings and all other forms of federal attacks.

But I think many scientific societies are too cautious, and have spent too much effort worrying about what the most reticent of their members would think. They will need their members to push them – hard – to take stronger actions.

Point 2: Yup. Many scientists are – or were – employees of the American government.

Points 3 and 4: I think these are variations of “Science isn’t political.” We saw this view promulgated late last year in an editorial by the head the National Academies of Science in the journal Science. I didn’t like it then and I think we have seen how badly misguided it was.

Point 5: Thankfully, I have not seen scientists pulling, “I got mine, so what?” I’m sure some exist, but I don’t think there are many.

I will add a sixth point. A few years ago, Randy Olson wrote:

Science is run by committees, top to bottom. Committees don’t lead, they facilitate. They don’t come up with good ideas and make them happen, they wait for individuals to come to them with good ideas that they can support or reject.

I’ve thought about this paragraph a lot. I don’t agree entirely that science is leaderless, but it surely tends to be run by committees. I don’t agree entirely that committees cannot come up with good ideas. But committees are slow and deliberative on purpose, and that is holding us back right now, where we need speed. 

That makes the role of strong existing “committees,” like scientific societies, all the more critical. Let’s not waste time striking new committees. Let’s mobilize the ones we have already.

If we can get them to be strong enough.

Related posts

Okay, stop. Saying “science isn’t political” will not keep science safe from political attacks 

Scientific societies are failing the moment by scrubbing diversity from their websites without explanation 

External links

Christina Pagel on Bluesky

Michael Crighton: The Lost Opportunity 

11 February 2025

These are not serious people

An American politician with a seat in the House of Representatives has introduced a resolution “to acquire Greenland and to rename Greenland as “Red, White, and Blueland”. 

This is real. Not a prank. Not a hoax. Not an imaginary story. Not an April Fools day joke.

External links

H.R.1161 - To authorize the President to enter into negotiations to acquire Greenland and to rename Greenland as "Red, White, and Blueland". 119th Congress (2025-2026)

Carter introduces bill renaming Greenland, February 11, 2025 


#ArsenicLife in soft focus

I am glad to hear that Felisa Wolfe-Simon is doing well and still loves science.

But lots of other information in a new article in the New York Times about Wolfe-Simon is not as warming.

First, the timeline of events circa 2010 described in the article is very fuzzy at best. I’m not sure if it6s wrong or just cloudy.

By focusing on Wolfe-Simon, others involved in this story – notably Rosie Redfield – are reduced to namely “critics.” There are other elements of the story that are maybe over emphasized?

Critique soon became attack, and attack often became personal – focusing, for instance, on Dr. Wolfe-Simon’s appearance, including her dyed hair.

Several articles described her colourful hair positively. And Rosie Redfield dyed her hair, too. I cannot doubt some attacked her appearance, and that any of those comments can hurt bad.

 And soon, Dr. Wolfe-Simon said, she couldn’t get grants or publish papers.

A lot of researchers struggle to get grants or publish papers. There’s no way of knowing what proposals or manuscripts she was submitting or what the comments were.

Second, there is news that the journal Science is considering retracting the original paper. Current editor, Holden Thorp, expands on the possible retraction of the arsenic life paper in a thread on Bluesky. i asked what point a retraction would serve, to which Thorp replied, “Partly because of AI and even before with search engines that only look at the title, papers like this continue to get cited uncritically. ‘Retraction’ is added to the title in these cases.”

This strikes me as reasonable, but I still have some reservations. By the same argument, anyone who only looks at the title and sees “Retraction” will also view that uncritically and not read the retraction notice. This could mean someone would not cite this paper appropriately for its discovery of a new strain of arsenic-tolerant bacteria (which is not contests). 

Retraction is a blunt instrument.

Third, this article paints Wolfe-Simon as someone who was wronged, not as someone who was wrong.

Usually, professional consequences so severe are reserved for those who fabricate data or commit fraud, which no one has alleged with #arseniclife. Why, then, were the repercussions so resounding for Dr. Wolfe-Simon?

And from reading this article, it seems to me that Wolfe-Simon still thinks she found bacteria that use arsenic instead of phosphorus.

She also defended the discovery against scientific consensus. Some see that as an unwillingness to change a conclusion in the face of new information. Dr. Wolfe-Simon would say that information is wrong.

And...

To this day, Dr. Wolfe-Simon defends the work, noting that she wishes the team had saved less data for a second paper. The team published a response to critiques in Science, and Dr. Wolfe-Simon disputes failed replications of their findings. Other co-authors say they also stand by the integrity of the original work.

What I would like to know if whether the co-authors stand by the conclusions of their work. Do they think they discovered a bacterium that uses arsenic in place of phosphorus?

Wolfe-Simon throws Science magazine under the bus a bit:

She wouldn’t pursue a flashy journal that would impose a heavy hand in publication and press, she said.

We absolutely have to make space for making mistakes and being wrong in science. But researchers have to be willing to admit those mistakes.

External links

Her discovery wasn’t alien life, but science has never been the same. (Paywalled.)
 

Subtleties of sex in sticklebacks

Three spined stickleback.
Daniel Bolnick wrote a thread on Bluesky about the biology of sex that I thought would make a great blog post. So I saved him the trouble and compiled it.

This has been lightly edited from the original thread.

Well, no surprise that (Jerry) Coyne used his blog (whyevolutionistrue) to argue with the SSE/SSB/ASN letter about definitions of sex. I’m not going to engage deeply with the letter here. Like any committee-drafted text it reflects compromises; I agree with some phrasing but some could have been clearer.

Instead, I want to respond specifically about his comments regarding my work on stickleback. He points out, quite correctly, that I have personally used the terms male and female, and sex, extensively in my own research on stickleback. True. 

But it misses the point.

Yes, stickleback can generally be categorized as male and female. And they have a XX female / XY male sex determination system (in some species, not in others). And yes, when I run stats I often check for effects of sex, and often find them. I’ve had whole papers devoted to sexual dimorphism. But Jerry might want to talk to me (or another sticklebacker) before making claims about stickleback sex differences. Because its not as clear cut as he likes. 

Yes, the large majority of stickleback I’ve caught are phenotypically male or female, in the sense that they have ovaries or testes. But I have caught quite a few stickleback (hundreds at least) with neither identifiable ovaries nor testes. I have twice seen stickleback with both (yeah, we dissect a lot of fish). These are rare, but they exist. In statistical analyses, these fish get coded as NA, and excluded.  

So when I publish on sexual dimorphism, or use male/female as a factor in my stats (which I do a lot), I am simplifying, setting aside a small minority to focus on overall tendencies of the majority.

That’s okay for stickleback, but its not ethical when applied to humans.

I could readily have published my papers instead with three levels of “sex”: male, female, and “unknown,” with the third group being comparatively small and under-powered for statistical purposes given my sample sizes. 

Then there’s the issue of trait differences between the sexes, which are clear-cut for most individuals in some environments, but less so in other environments. Stickleback are famous for males having bright red chins and blue eyes as a sexual signal... except when they don’t. There are plenty of lakes where I work in BC, in which males are dull brownish-green, and  indistinguishable visually from females. Indeed, in some lakes we have a hard time generating in vitro crosses because we’ll grab a fish we think is male, only to find after we euthanize and dissect it to obtain sperm, that it had ovaries.

For many traits that we have measured (diet, isotopes, morphology, size, color, immune traits, gene expression, metabolomics), males and females are on average different. But for most of these traits the sexes distributions overlap extensively. The magnitude of that dimorphism varies. In some lakes it can be easy to distinguish the sexes (for most individuals), other lakes it can be very hard, even in multivariate trait space. We can measure this dimorphism using things like linear discriminant function analysis (LDA). This LDA can score individuals as to how distinctively male or female they are and it varies numerically in ways we can measure precisely. Some populations are more dimorphic, others less. These differences persist in the lab, meaning there is genetic variation in between-sex differences.

To conclude: yes, I use the terms male and female in my work, reflecting real features of the fish I study. But I also set aside individuals who don’t fit neatly into the categories; again, that's a fair simplification when working with fish, but not morally defensible with people. 

I also emphasize that even after 25 years studying stickleback, I can mis-identify an individual’s sex, even when the gonads are unambiguous. In these cases the gametic sex is a real distinction, but its relation to other traits is complex.

I'm not personally a deep expert on the biology of sex, that's not my specialty. But, if Coyne wants to have a conversation about sex and sexual traits in stickleback, I'd be happy to point out the subtleties. Most of all, the subtlety that seems to elude him is we can recognize male/female differences that characterize many individuals, and use this distinction, yet still acknowledge that these differences do not apply completely to fully 100% of a population.

Anisogamy (Differences in size of gametes - ZF) will surely come up in comments. I don’t have a problem with this criterion as a broad rule, but again there exist exceptions that make it less than universal. Individuals exist that produce no gametes: genetic sterility, environmentally induced sterility, parasitic castration. For instance, the parasite I study, Schistocephalus, can completely sterilize its host in some cases: we find males with no functional testes, females with non-functional ovaries. Do they cease to be males/females because they don’t have any gametes (whether large or small)? 

To wrap up at last: my impression as an empiricist specializing on a particular fish species: stickleback sex is clearly delineated most of the time (hence, I use the term a lot), but not in every single instance. 

The latter is where Jerry misses the point: Once we pivot into the world of humans, the biology of sex in stickleback (and how I simplify it slightly in my work) is irrelevant to the ethical standards by which we treat each other.

Update, 21 February 2025: Nathan Lents provides another great thread on the biology of sex. (Lightly edited.)

Biological sex and sex differences are real, but poorly captured by a simple binary.

No one is saying there is no such thing as “male” and “female.” (Of course there is.)

No one is saying there aren’t difference between “male” and “female.” (Of course there are.)

No one is saying that “male” and “female” aren’t real categories. (Of course they are.)

We’re saying that these categories are continuous. Continuous categories are still categories. There are many examples of continuous categories in biology, but the simplest is developmental age. Child and adult are real categories, with real and important differences. A child and an adult are different and must be treated as such. But there is no hard line we can draw between those two categories. It’s a spectrum, and a gradual transition. They are continuous categories. But that doesn’t mean the categories aren’t real and important.

There are borderline cases where it’s not simple to say which category someone belongs in – child or adult – but that doesn’t mean the categories are meaningless.  All of adolescence could be seen as fuzzy gray area. Child and adult are still different.

Dawkins and Coyne’s “gotcha” moments of finding authors using the term “male” and “female” in their publications isn't the flex you think it is. They (and you!) probably use other continuous categories like adult, juvenile, adolescent. Continuous categories are still categories.

The same is true for speciation, of course. There wasn’t a Homo heidelbergensis woman that gave birth to a H. sapiens baby. (See? I used “woman” and no one will take issue!) The transition between species is gradual and continuous, a spectrum, but species are still different.

And this is what we’re saying with biological sex. The categories are not as easy to define as we thought because almost all sex differences are bimodal, not binary. Except for the gametes (I’ll get to them in a sec), pretty much all sex differences are continuous.

Examples:

Height, muscle mass, hormone levels & ratios, red blood cell count, white blood cell count, basal metabolic rate, bone density, thickness of the corpus collosum, ratio of white matter and gray matter, size of  pre-optic area, breast size, facial hair, waist-hip ratio, libido… These are all features of sexed bodies that show continuous bimodal sex differences. Don’t all of those matter a whole lot more to health, daily life, relationships, than what gamete you make? Yes, gametes are binary, but they also matter very little most of the time.

There are important health differences. Women are nine times more likely to get lupus. Men two and a half times more likely to get Parkinson’s. Women suffer depression more. Men die by suicide more. Women more often anemic. Men more heart attacks. These are important, but have nothing to do with gametes. There are two liver diseases that are, bizarrely, reciprocally bimodal: Women are nine times more likely to get primary biliary cirrhosis cases, while men are nine times more likely to develop primary sclerosing cholangitis.

Recognizing that biological sex is complex doesn’t deny sex differences, it affirms them. I understand why some want to center the gametes – they are simple and binary. But that ignores all kinds of important sex differences all over the body that actually matter. In my view, the fact that people are retreating to gametes to define sex shows that they know that all of the other sex differences are continuous / bimodal.  

Men and women are real categories with real differences, but the categories are continuous and differences are bimodal.

And I haven’t even gotten to intersex people, who are real people that exist. They are almost always healthy, usually fertile, and in any case deserving of dignity and respect. They, too, cannot be ignored or discarded, simply out of preference for nice neat categories.

Biology is messy, adaptable, and endlessly creative. It rarely traffics in binaries and loves wondrous variety. Life will always frustrate the human desire for neat categories with simple, universal definitions. Diversity is a universal feature of life.  Let's acknowledge celebrate that!

Photo by Jason Ching/University of Washington on Flickr. Used under a Creative Commons license.

10 February 2025

Scientific societies are failing the moment by scrubbing diversity from their websites without explanation

I called last week, “The worst week for science ever.” Of course, it was just the worst week so far.

This morning, I saw reports of two large scientific societies – The American Chemical Society (ACS) and the American Geophysical Union (AGU) – editing their websites to remove or reduce mentions of diversity. 

“The ACS has deleted its website on Diversity, Equity, Inclusion & Respect.” – David K. Smith

“The main DEI page from AGU has been modified to take diversity out of the title.” – Celeste Labedz

This is on top of the American Society for Microbiology removing a bunch of pages relating to diversity and replacing them with “Under review.” 

Not all societies are taking this road. The Society for the Study of Evolution released an message today, for instance, said the society still supported “community building in an equitable and globally inclusive manner.” Near the end, it concludes, “we note that the attacks on historically excluded members of society are reprehensible. We encourage our membership to be unwavering in your support of the most vulnerable within the community.”

My hypothesis about these is that somewhere in these big societies, there is one or more federal grants supporting some program or another.  They had lawyers review the executive orders, who suggested that they change the website to avoid getting sued. A lawsuit exposes the entire society to financial risk, or maybe even ruin.

I appreciate that scientific societies want to continue to exist and are sensitive to risk.

But I am deeply disappointed that there seems to be no communication about these changes. Words and phrases and entire pages are disappearing with no warning or notice. That’s not transparency and it sure as anything isn’t leadership.

07 February 2025

Worst week for science ever

Sunday: American Society for Microbiology scrubs mentions of diversity from their websites.

Tuesday: NASA stops everything to scrub mentioned of diversity from their websites; ICE arrests a student on campus; NSF announces plans to make deep cut to staff; USAID is halted.

Wednesday: HHMI kills one of its diversity program; NIH trashes applications for diversity F grants.

Friday: NIH cuts its indirect costs and sets them to 15%.

Yes, these are all American centric. But the United States was, until the start of 2025, the world’s undisputed science superpower. These effects will not be contained within the borders of the United States.

05 February 2025

Untitled post 2025-02-05

The United States has doused itself in gasoline and lit a match.

The only question is whether the fire can be put out with no more than scorched clothes, or whether it will burn until even the bones are ash.