17 March 2016

A pre-print experiment: will anyone notice?

In late February, there was a lot of chatter on my Twitter feed from the #ASAPBio meeting, about using pre-prints in biology.This has been the accepted practice in physics for decades.

My previous experience with pre-prints was underwhelming. I’d rather have one definitive version of record. And I’d like the benefits of it being reviewed and edited before release. Besides, my research is so far from glamorous that I’m not convinced a pre-print makes a difference.

Following the ASAPbio meeting, I saw congratulatory tweets like this:

Randy Schekman strikes again: yet another #nobelpreprint - Richard Sever
Marty Chalfie on bioRxiv! That’s Nobel #2 today - Richard Sever
Yay, Hopi Hoekstra (@hopihoekstra) just published on @biorxivpreprint - Leslie Voshall

Similarly, a New York Times article on pre-prints that appeared several weeks later focused on the Nobel laureates. I admit I got annoyed by tweets and articles about Nobel winners and Ivy League professors and HHMI labs and established professors at major research universities using pre-prints. I wasn’t the only one:

 I wish this article didn’t erase the biologists who have been posting to arXiv for years.

If pre-prints are going to become the norm in biology, they can’t just work for the established superstars. Pre-prints have to have benefits for the rank and file out there. It can’t just be “more work.”

For example, I think one of the reasons PLOS ONE was a success was that it provided benefits, not just for superstars, but for regular scientists doing solid but incremental work: it provided a venue that didn’t screen for importance. That was a huge change. In contrast, new journals that try to cater to science superstars by publishing “high impact” science (PLOS Biology or eLife and Science Advances), while not failures, have not taken off in the same was that PLOS ONE did.

I decided I would try an experiment.

I don’t do the most glamorous scientific research, but I do have a higher than average social media profile for a scientist. (I have more Twitter followers than my university does.) So I thought, “Let’s put up a pre-print on biorXiv and see if anyone comments.”

I spent the better part of a morning (Thursday, 25 February 2016) uploading the pre-print. Since I had seen people whinging about “put your figures in the main body of the text, not at the end of the paper,” I had to spend time reformatting the manuscript so it looked kind of nice. I also made sure my Twitter handle was on the front page, to make it easy for people to let me know they’d seen my paper.

I was a little annoyed that I had to go through one of those clunky manuscript submission systems that I do for journals. I had to take a few stabs at converting the document into a PDF. biorXiv has a built-in PDF conversion built into it, but the results were unsatisfactory. There were several image conversion problems. One picture looked like it came out of a printer running low on ink. Lines on some of the graphs looked like they had been dipped in chocolate. Converting the file to PDF on my desktop looked much better. I uploaded that, only to find that even that had to go through a PDF conversion process that chewed up some more time.

biorXiv preprints are vetted by actual people, so I waited a few hours (three hours and thirty-nine minutes) to get back a confirmation email. It was up on biorXiv within a couple of hours. All in all, pretty quick.

I updated the “Non-peer reviewed papers” section of my home page. I put a little “New!” icon next to the link and everything. But I didn’t go out and promote it. I deliberately didn’t check it on biorXiv to ensure that my own views wouldn’t get counted. Because the point was to see whether anyone would notice without active promotion.

I waited. I wasn’t sure how long to wait.

After a day, my article had an Altmetric score of 1. biorXivpreprints and three other accounts that looked like bots tweeted the paper, apparently because they trawl and tweet every biorXiv paper. (By the way, “Bat_papers” Twitter account? There are no bats in my paper.) The four Twitter accounts combined had fewer followers than me. Looking at the Altmetrics page did remind me, however, that I need to make the title of my paper more Twitter friendly. It was way longer than 140 characters.

Four days later (29 February 2016), I got a Google Scholar alert in my inbox alerting me to the presence of my pre-print. Again, this was an automated response. That was another way people could have found my paper.

Three weeks has gone by now. And that’s all the action I’ve seen on the pre-print. Even with a New York Times article brought attention to pre-prints and biorXiv, nobody noticed mine. Instead, the attention is focused on the “established labs,” as Arturo Casadevall calls them. The cool kids.

I learned that for rank and file biologists, posting work on pre-prints is probably just another task to do whose tangible rewards compared to a journal article are “few to none.” Like Kim Kardashian posting a selfie, pre-prints will probably only get attention if a person who is already famous does it.

Update, 18 March 2016: This post has been getting quite a bit of interest (thank you!), and I think as a result, the Altmetric score on the article reference herein has jumped from 1 to 11 (though mostly due to being included in this blog post).

Related posts

The science of asking
Mission creep in scientific publishing


Faulkes Z. 2016. The long-term sand crab study: phenology, geographic size variation, and a rare new colour morph in Lepidopa benedicti (Decapoda: Albuneidae) biorXiv. http://dx.doi.org/10.1101/041376

External links

The selfish scientist’s guide to preprint posting
Handful of Biologists Went Rogue and Published Directly to Internet
Taking the online medicine (The Economist article)

Picture from here.


Mike Taylor said...

Well, I can't agree with much of that.

I do share your distaste for the lionising of Nobel laureates. I don't think it's a good way of measuring what really matters and doesn't in science. But surely the point of repoerting Nobellists' use of preprint archives is "Look, this isn't just a shiny toy for kids."

As for your experiment: it's interesting, but artificial. The purpose of preprinting is not as a means to get more publicity for your paper, but to make it available. All you've really done here is show that you don't get what would have been a pleasant bonus side-effect.

practiCal fMRI said...

I second Mike's comments. Most importantly re. the coverage, if the Big Cheeses don't do it then the masses down at our level will be unlikely to want to shift, so in that sense the coverage can be seen as a Massive Step Forward.

A more useful (but similarly anecdotal) experiment would be to measure the time it takes to make a preprint available and useful to colleagues and your field, versus the time it takes for that very same (initial) manuscript to see the light of day in a peer-reviewed journal. Then you could try to quantify the additional science done while the manuscript languished in the trad pipeline.

Unknown said...

I think that the point was more that pre-prints will only work if rank and file researchers do see that pleasant bonus side-effect. Otherwise, it will just be another chore that the majority of researchers won't bother with.

practiCal fMRI said...

Hi Lisa, I'm file if not rank, and I have already experienced the benefits of preprints. As Mike indicated, the tool is for making work available (rapidly), not publicizing it.

Mike Taylor said...

You can't get ranker or more file than I am -- I'm not even a proper postdoc, just a Research Associate. I can tell you that my own experience of preprinting has been hugely positive. I've written about Four different reasons to post preprints, and I certainly plan to do it again.

Mark Wilson said...

Another important reason to post preprints to a recognized server is to establish priority, in case someone publishes the same thing later in a journal while yours is languishing in the journal pipeline.

Mark Wilson said...

I have never received any direct response from anyone when my paper is published in a journal. But I have received it when posting on arXiv.org.