18 April 2014

The varieties of peer review

I’ve been very pleased with the response to my Neuron article on post-publication peer review. (Read more about the backstory here). The commentary on PubPeer got me thinking some more about the different kinds of peer review. In particular, the topic of pre-print servers like arXiv came up. I noted that arXiv is “moderated by experts”... which sounds kind of like peer review.

This lead me to thinking about the continuum from traditional pre-publication peer review to post-publication peer review, from “sanity test” to predicting “impact” or “importance.” Before you know it, I made this (click to enlarge):

Everyone wants peer review to be thorough and rigorous, so that’s the horizontal X axis. Because my last article was on post-publication peer review, the time relative to widespread distribution is the vertical Y axis.

Some points (like the two points for “Conference” and “Peer review”) are separated in the vertical Y axis to make the text legible, not because one always happens before the other. Likewise, each point should be a smear indicating a range, not a point. But this is just a first pass. Maybe this can help characterize the different kinds of peer review and commentary.

Additional: To answer a question posed by Jason Goldman, by “blog posts,” I had in mind blog posts by scientists.

Related posts

Kabuki theatre versus the mosh pit: notes from my Neuron article on post-publication peer review


Anonymous said...
This comment has been removed by a blog administrator.
Thomas Arildsen said...

Good idea. A nice next step would be to make the plot a survey, allowing respondents to adjust the point locations, and display the results as probability density "clouds".

Bonnie Swoger said...

As a librarian, one of my jobs is to help students evaluate the information sources they find. I really like the simplicity of the diagram, as it helps to understand different types of evaluation. It also helps to emphasize that published scholarly work really goes through a wide variety of evaluations. Thanks!

Mike Taylor said...

A couple to add:

Comments on a pre-print (such as the super-helpful ones on our PeerJ Barosaurus pre-print which amount to peer-reviews).

Comments on a blog about a paper. These sometimes contain real meat.

Re-use of data published alongside a paper.