Recently Daniel MacArthur pointed to the vibrant discussion over at Genomes Unzipped on a moderately infamous paper from Science last year, Widespread RNA and DNA Sequence Differences in the Human Transcriptome, asserting that it is “exactly what open peer review should be like.” This made me wonder, it’s been over five years since Chris Surridge asked why there was so much more commentary on a PLoS ONE paper, By Hook or by Crook? Morphometry, Competition and Cooperation in Rodent Sperm, on blogs than on the paper itself. Has anything changed? The most viewed paper on PLoS Biology, How Many Species Are There on Earth and in the Ocean?, has 9 comments for 45,000 article views. In contrast, Genomes Unzipped has 14 comments for likely far fewer page views. Additionally, if you find the post on the weblog the comments automatically load. Not so with the PLoS Biology paper, you have to click through (yes, I see how this can be a feature, not a bug, but in that case why even bother with comments if you provide an email address for correspondence?)
I’m not going to rehash Joseph Pickrell’s argument in Why publish science in peer-reviewed journals? Rather, I want to reiterate that the ultimate goal here is to figure out how the world works. To understand that you need to get a sense of the science, and if you aren’t a specialist simply consulting peer reviewed literature is not always sufficient or practicable. Recently a researcher was discussing the Science paper at issue above to some graduate students, reiterating Joseph Pickrell’s points. When someone was curious about further details the individual explaining the controversy specifically cited Pickrell’s earlier post on the topic in Genomes Unzipped to get an understanding of the broader issues of the critique. From what I know this person is not a major reader of science blogs. Rather, Pickrell’s post likely came up high on a search engine query, and to his mind was an appropriate reflection of the scientific consensus as he understood it. The question is why does the discussion not begin in the journals themselves? Why do curious people have to scour the internet? When a controversy or “big paper” emerges in a specific area outside of my detailed focus I know what I do: I simply load the appropriate science blogs which cover that topic. There I hope to get a sense of what the people in the field think. Peer review is not sufficient in many cases. Science is a production of humans, and so is fraught with politics, and reputations are thrown about to get a paper in “high impact” journals.
But is anything broken in the first place? Is this how it’s supposed to work, and how it’s going to work? Journals make a huge argument for their value. They charge academic libraries quite a mint (with honorable exceptions made for open access journals!). It seems that they should make at least a good faith effort at post-publication review and extension. Did you know that Nature has a comments feature? Have you ever seen a comment posted via this feature?