“Pay reviewers” has become a social media slogan that could be a welcome change to academic publishing if potential pitfalls are avoided. But because we are researchers, we need to do the experiment. A recent paper by Cotton and colleagues (2025) is the first I have seen to test what paying peer reviewers might accomplish. (There was a prior test by Chetty and colleagues in 2014.)
The journal Critical Care Medicine offered reviewers an incentive of $250 on alternate weeks.
Does offering to pay peer reviewers get more reviews? The authors measure this in three ways.
- Percent of invitations accepted. There was no difference between offering payment or not.
- “Rate of conversion,” that is, the percent of reviews actually submitted. Offering an incentive improved this measure.
- “On time rate of conversion,” which was the percent of reviews submitted within two weeks. This is probably the most relevant measure for a journal, since timely review is important. This was again improved by offering an incentive: about 42% of invitations got a review back in two weeks when offered the incentive, compare to about 32% without. The average difference in time to return a review is a day.
By the way, for those 32% of reviewers who got a review done within two weeks for no pay, I appreciate you. Not all heroes wear capes.
Is peer review quality affected? Not noticeably. The results note that about a third of the reviews that were received did not got a quality assessment. By reading between the lines, it appears that the handling associate editors were only “encouraged” to grade the reviews, and a chunk of the editors just did not rate the reviews.
There is a difference by offering payment, but is it worth it? Trevor Branch noted:
(T)hese results (8% higher rate of completing review, 1 day earlier on average) don't exactly motivate journals to pay reviewers.
Cotton and colleagues notes that the costs add up.
(F)or a journal with 1,000 reviewed submissions per year, three paid reviewers per article, and payments of $250 per reviewer, such a payment mechanism could cost $750,000 annually.
Even if you ramped it down to two reviewers, which is common for many basic science journals, it’s not clear that the cost-benefit ratio of paying peer reviewers is worth it for journals.
This study can’t address many of the other possible problems that will undoubtedly emerge. For example, what will authors say if they found if their article is rejected on the basis of what they think is a shoddy review that the journal paid for? What is an appropriate amount of incentive?
I recognize that for many, paying for peer review is about moral fairness, not financial prudence.
On a related note, Stat News has an update on the lawsuit launched against scientific publishers last year. The publishers are asking for a dismissal for similar reasons that I outlined last year.
Legal experts told STAT that while there’s little question unpaid peer review has helped enrich publishers, it may be difficult to prove the practices cited in the lawsuit were the product of an anticompetitive agreement.
Much of the article, however, is as much about about the service model of peer review as the lawsuit itself. The service model of peer review is under strain, but I don’t think the article articulates the reasons why very well.
First and foremost, the assessment schemes of most research institutions are weighed far too heavily towards output.
Second, the globalization of science means that the community of peers is more diffuse. Peere review works best, I think, when many people know each other.
Third, there are likely still imbalances in who is asked to review papers (e.g., established researchers from established institutions in countries with colonial histories are asked more often than researchers everywhere else).
References
Chetty R, Saez E, Sándor L. 2014. What policies increase prosocial behavior? An experiment with referees at the journal of public economics. Journal of Economic Perspectives 28:169–188. https://doi.org/10.1257/jep.28.3.169
Cotton CS, Alam A, Tosta S, Buchman TG, Maslove DM. Effect of monetary incentives on peer review acceptance and completion: A quasi-randomized interventional trial. Critical Care Medicine: in press. http://doi.org/10.1097/CCM.0000000000006637
No comments:
Post a Comment