15 November 2016

Peer review pariah, update

One of the good things about having a long running blog is that you rediscover stuff you wrote and can update it.

Back, um, “some time ago” (six years now crap I’m old), I dealt with the question of whether peer reviewers are overburdened. That is, there are too many papers and not enough people willing to review them all. At the time, I was suspect of the claims that being asked to review one to three papers a month was normal.

This question came up again on Twitter today. Since it’s been a few years, I wondered if I was still a pariah. I got the impression I was being asked to review more now...

And I am. The trendline is definitely upward. But it’s still far less than the “one to three papers a month” figure that people were claiming. I might hit the “one per month” around 2022.

I was part of the reviewer “talent pool” in the early 2000s, but got very few invites. I am the same guy now as then, so what’s changed? I think I’m getting asked to do more reviews because of the time spent in the academic system. And I was, luckily, able to step up my own publication game around 2010, which may have contributed to my “name visibility” among editorial boards.

I know some people on Twitter who are on journal editorial boards, and they do indeed complain about finding reviewers. But I wonder how well editors use the available talent pool. I would bet that journal peer review invitations are biased against:

  • Faculty who are not at American universities. (Update, 21 November 2016: Warne (2016) reports proportionately more peer review is performed by American researchers than Chinese ones.)
  • Faculty who are not at English-speaking universities.
  • Faculty at undergraduate institutions.
  • Post-doctoral fellows and graduate students.
  • Women reviewers.
  • Minority reviewers.

Update, 21 November 2016: Table 1 in Okike et al. (2016) shows more than ten men for every one woman reviewing manuscripts. Hat tip to Laura Jurgens.

Update, 5 December 2016: This tweeted list of “Top reviewers” from the journal Neurospsychopharmacology has nine men and one woman. Hat tip to Bita Maghaddam.

Update, 23 March 2017: Big new study in eLife by Helmer and coleagues (2017) supports that the hypothesis that “women are underrepresented in the peer-review process.” This comes on the heels of a Nature article that also supports this hypothesis.

Update, 2 May 2017: Biochem Belle pointed out that Fox et al. (2017) showed that women were much, much less likely to be suggested by authors as reviewers. The highest year as only 25% in 2014.


Fox CW, Burns CS, Muncy AD, Meyer JA. 2017. Author-suggested reviewers: gender differences and influences on the peer review process at an ecology journal. Functional Ecology 31: 270–280. http://dx.doi.org/10.1111/1365-2435.12665

Helmer M, Schottdorf M, Neef A, Battaglia D. 2017. Gender bias in scholarly peer review. eLife 6: e21718. http://dx.doi.org/10.7554/eLife.21718

Lerback J, Hanson B. 2017 Journals invite too few women to referee. Nature 541(7638): 455–457. http://dx.doi.org/10.1038/541455a

Okike K, Hug KT, Kocher MS, Leopold SS. 2016. Single-blind vs double-blind peer review in the setting of author prestige. JAMA 316(12): 1315-1316. http:/dx.doi.org/10.1001/jama.2016.11014

Warne V. 2016. Rewarding reviewers – sense or sensibility? A Wiley study explained. Learned Publishing 29(1): 41-50. http://dx.doi.org/10.1002/leap.1002

Related posts

Peer review pariah

1 comment:

practiCal fMRI said...

I've always tried to review between 2x and 3x the number of papers I'm publishing per year. I figure that's a fair ratio. (In part because one of my pet peeves is someone who thinks they should be on twenty papers a year and then claims everyone's too busy to review!) But of late I've been running into a problem: namely that I only have meaningful expertise in a small part of a complex, interdisciplinary endeavor. So I end up having to say no a lot purely because I know I won't be able to do most of a manuscript justice.

I made a suggestion that would help me in my field: http://blogs.plos.org/neuro/2015/02/25/all-or-nothing-versus-partial-peer-review/ So far, though, nobody has bought it for pre-publication review. As I mentioned in the post, I can still conduct partial peer review using PubMedCommons or PubPeer, but I am still not averse to helping out pre-pub, if those parts can be disentangled from the bits I have no clue about.