Maliniak and colleagues (2013) have found an important fact: if a paper in international relations has only female authors, it will be cited about 70% as often as a paper with only male authors.
In a
guest blog post, co-author on that study, Barbara Walter, says there is a “relatively easy” solution to fix this. I was reminded of the saying, “For every complex problem there is a simple solution, and it’s wrong.”
(There’s a) potentially easy solution: anonymity. What if we set up evaluations in academia so that we never knew the gender of the person being evaluated (or at the very least downplayed it as much as possible)?
Under this suggestion, I should be “Z. Faulkes,” always, in my academic writing.
I am sympathetic to the idea of removing identifiers. After all, if something is supposed to be judged on its content, then let’s focus on the content. But what bugs me is that the authors say this is “relatively easy.” Indeed, the example they use makes it
sound easy:
Prior to adopting anonymous auditions in which candidates sat behind screens when they played, less than 5 percent of all musicians in these orchestras were women. Once blind auditions were instituted women were 50 percent more likely to make it out of the preliminary rounds and significantly more likely to ultimately win the job. Today, 25 percent of all musicians in these elite orchestras are women.
The more I thought about it, the more the analogy with a musical audition unraveled.
A musical audition asks different performers to perform a set, predetermined piece of music. In academic evaluation, researchers are submitting original works, and no two are alike.
A musical audition is a solo endeavor, where you can judge the performance of the individual. In academics, particularly in science, you are often working with many authors, and it’s very difficult to disentangle the contributions of the authors.
Evaluating academics isn’t like auditioning a single soloist in an orchestra. It’s more like digging through demo tapes of rock bands. And you may have heard the band before at a bar or showcase gig.
For academics, their manuscripts and their CVs are extensively reviewed.
Double blind reviews of single manuscripts has been tested occasionally. For instance. Budden and colleagues (2008) claimed this increased the number of papers with women as the first author. Those results were contested, however (Webb et al 2008; Whittaker 2008). But even the authors of the original study admitted it was incredibly difficult to get data on the effects of different peer-review practices, since so few journals are willing to tinker with it, or share data about it.
As a practical matter, I don’t think there is any way to make a CV
even close to anonymous. Manuscripts are perhaps a more promising target
for being reviewed anonymously. The nature of peer review, trying to
find the most qualified people to review papers with the most expertise,
makes it highly likely that reviews will be done by people who have met
the authors and know what their research is.
For instance, early in my career, I got wind that my name had come up in an Animal Behavior Society meeting. Someone (I think it was Zuleyma Tang-Martinez) was making the point that the society should know the sex of all of its members, “For instance, is this ‘Zen Faulkes’ male or female?”
As it happened, not one, but
two of my professors from my university were in the room. They laughed, and said, “Male... You should have asked!”
For that matter, making a work anonymous will most likely change the
“females cited 70% as often as males” if the problem is bias at the
reviewing stage. It is unlikely to address the citation problem that is
coming post review, like males being more active in self-promotion and
networking.
A few years ago at Science
Online, Ed Yong mentioned that he often had scientists emailing him,
asking if Ed could mention their newest work. These were invariably men.
The women in the room reacted with surprise. None of them had thought
that this was something that they could or should do. Whether you think such requests are out of line or not,
you could see how that kind of difference in promotion could lead to men
being better known in their field, and getting some more citations as a
result.
That the scientific community is a community means that anonymity will be difficult to come by except in very early careers. Double blind review might be much more practical in situations like applications to grad school.
Let’s fight biases whenever we come across them. but let’s not pretend it’s going to be easy. It’s going to be a tough, hard slog.
Hat tip to Miriam Goldstein for bringing this to my attention.
Update, 16 May 2015: Any discussion of making peer review blinded or not is probably incomplete without referencing
this post by Hilda Bastian, which systematically reviews a big chunk of the literature on effects of anonymity in peer review. She
concludes:
I think institutionalizing anonymity in publication peer review is
probably going out on a limb. It’s only partially successful at hiding
authors’ identities, and mostly only when people in their field don’t
know what authors have been working on. If blinding authors was a
powerful mechanism, that would be evident by now.
Update, 21 September 2022: Another paper claiming to show bias in peer review made the rounds recently on Twitter, which again prompted many people to argue, “Blinding manuscripts is the answer, obviously.” Hilda Bastian once more enters the fray with another deep read and analysis of the paper. A few choice points from her post.
An effect on peer reviewers’ opinions isn’t the same thing as an effect on publication bias.
Good editors matter.
Could this manuscript be an inherently divisive one? We can’t know,
because no evaluation of the manuscript by methodological and content
experts was reported, and the expertise of the peer reviewers for the
particular subject matter wasn’t reported either.
We are being asked to take a lot on trust here.
Hilda concludes that this paper is still relatively unconvincing in making a case for blinded peer review. She notes that a big study is in the works and should be out soon. But I recommend reading her whole post.
References
Maliniaka D, Powers R, Walter BF. 2013. The gender citation gap in international relations. International Organization 67(04): 889-922. DOI: http://dx.doi.org/10.1017/S0020818313000209
Budden AE, Tregenza T, Aarssen LW, Koricheva J, Leimu R, Lortie CJ. 2008. Double-blind review favours increased representation of female authors. Trends in Ecology & Evolution 23(1): 4-6. http://dx.doi.org/10.1016/j.tree.2007.07.008
Webb TJ, O’Hara B, Freckleton RP. 2008. Does double-blind review benefit female authors? Trends in Ecology & Evolution 23(7): 351-353. http://dx.doi.org/10.1016/j.tree.2008.03.003
Whittaker RJ. 2008. Journal review and gender equality: a critical comment on Budden et al. Trends in Ecology & Evolution 23(9) 478-479. http://dx.doi.org/10.1016/j.tree.2008.06.003
Budden AE, Aarssen LW, Koricheva J, Leimu R, Lortie CJ, Tregenza T. 2008. Response to Whittaker: challenges in testing for gender bias. Trends in Ecology & Evolution 23(9): 480-481. http://dx.doi.org/10.1016/j.tree.2008.06.004
External links
How to reduce the gender gap in one (relatively) easy step
The problem is that scientists are human
Weighing up anonymity and openness in publication peer review
Photo by Penn Provenance Project on Flickr; used under a Creative Commons license.
I went to a Doctor Who convention in 1986. John Nathan-Turner, then producer of the show, was asked about efforts to recover the old missing stories. He said something like, “They may have dried up, and there may be no more to find.”
