The use and misuse of journal Impact Factors continues to attract attention. I think Impact Factor has some usefulness in showing a journal is a bona fide scientific enterprise.
In my non-scientist life, though, I’ve seen other situations where people are aggressively pursuing some stat or another to prove their worth. When I was involved in the card game Legend of the Five Rings, there was a point where people were repeatedly calling for the game to use the Elo system to determine who were the best players.
Elo is not an acronym, but refers to Arpad Elo, who devised a system for ranking chess players. Like Impact Factor, it has sway. Players take Elo rankings pretty seriously for games where it is used. Many card gamers knew it from Magic: the Gathering.I like this quote from Arpad Elo (written as part of a Chess Life article in 1962), because it showed Elo understood the problems of uncertainty well.
Often people who are not familiar with the nature and limitations of statistical methods tend to expect too much of the rating system. Ratings provide merely a comparison of performances, no more and no less. The measurement of the performance of an individual is always made relative to the performance of his competitors and both the performance of the player and of his opponents are subject to much the same random fluctuations. The measurement of the rating of an individual might well be compared with the measurement of the position of a cork bobbing up and down on the surface of agitated water with a yard stick tied to a rope and which is swaying in the wind.
Emphasis in the original.
Interestingly, after Legend of the Five Rings did adopt the Elo rating system, it did not prove to be “the answer” that some players thought it would be.
Picture of Elo from here.
2 comments:
Interesting analogy, makes me wonder if it can be pushed further. For instance, in chess the Elo system appears to be subject to rating inflation. Ratings at the highest level have been trending upward, at a faster rate than seems consistent with overall improvement in chess playing ability. I'm curious if something similar happens with impact factors...
My understanding is that Impact Factors have been increasing across the board. This makes sense to me. There are more scientists publishing in more journals and generating more citations. I can’t see how they could not go up. And I don’t see Thomson releasing an inflation index.
Post a Comment