Peer review pariah, update

One of the good things about having a long running blog is that you rediscover stuff you wrote and can update it.

Back, um, “some time ago” (six years now crap I’m old), I dealt with the question of whether peer reviewers are overburdened. That is, there are too many papers and not enough people willing to review them all. At the time, I was suspect of the claims that being asked to review one to three papers a month was normal.

This question came up again on Twitter today. Since it’s been a few years, I wondered if I was still a pariah. I got the impression I was being asked to review more now...

And I am. The trendline is definitely upward. But it’s still far less than the “one to three papers a month” figure that people were claiming. I might hit the “one per month” around 2022.

I was part of the reviewer “talent pool” in the early 2000s, but got very few invites. I am the same guy now as then, so what’s changed? I think I’m getting asked to do more reviews because of the time spent in the academic system. And I was, luckily, able to step up my own publication game around 2010, which may have contributed to my “name visibility” among editorial boards.

I know some people on Twitter who are on journal editorial boards, and they do indeed complain about finding reviewers. But I wonder how well editors use the available talent pool. I would bet that journal peer review invitations are biased against:

• Faculty who are not at American universities. (Update, 21 November 2016: Warne (2016) reports proportionately more peer review is performed by American researchers than Chinese ones.)
• Faculty who are not at English-speaking universities.
• Faculty at undergraduate institutions.
• Post-doctoral fellows and graduate students.
• Women reviewers.
• Minority reviewers.

Update, 21 November 2016: Table 1 in Okike et al. (2016) shows more than ten men for every one woman reviewing manuscripts. Hat tip to Laura Jurgens.

Update, 5 December 2016: This tweeted list of “Top reviewers” from the journal Neurospsychopharmacology has nine men and one woman. Hat tip to Bita Maghaddam.

Update, 23 March 2017: Big new study in eLife by Helmer and coleagues (2017) supports that the hypothesis that “women are underrepresented in the peer-review process.” This comes on the heels of a Nature article that also supports this hypothesis.

Update, 2 May 2017: Biochem Belle pointed out that Fox et al. (2017) showed that women were much, much less likely to be suggested by authors as reviewers. The highest year as only 25% in 2014.

Updated, 10 September 2017: Matt Hodgkinson tweets that for geology journals, women are underrepresented as suggested reviewers, and decline more often than men. But it is getting better.

References

Fox CW, Burns CS, Muncy AD, Meyer JA. 2017. Author-suggested reviewers: gender differences and influences on the peer review process at an ecology journal. Functional Ecology 31: 270–280. http://dx.doi.org/10.1111/1365-2435.12665

Helmer M, Schottdorf M, Neef A, Battaglia D. 2017. Gender bias in scholarly peer review. eLife 6: e21718. http://dx.doi.org/10.7554/eLife.21718

Lerback J, Hanson B. 2017 Journals invite too few women to referee. Nature 541(7638): 455–457. http://dx.doi.org/10.1038/541455a

Okike K, Hug KT, Kocher MS, Leopold SS. 2016. Single-blind vs double-blind peer review in the setting of author prestige. JAMA 316(12): 1315-1316. http:/dx.doi.org/10.1001/jama.2016.11014

Warne V. 2016. Rewarding reviewers – sense or sensibility? A Wiley study explained. Learned Publishing 29(1): 41-50. http://dx.doi.org/10.1002/leap.1002

Related posts

Peer review pariah