By Dennis D. McDonald

In Stop deifying “peer review” of journal publications Jonathan Eisen skillfully walks the reader through three realities of how peer review works with scientific journals:

1. Peer review is not magic
2. Peer review is not binary
3. Peer review is not static

In other words, peer review, where a prospective journal author’s “peers” are asked by a journal editor, often confidentially, to review and comment on manuscripts submitted for possible publication, is not perfect.

Eisen reiterates often quoted criticisms of peer review which I won’t repeat here. He emphasizes that peer review is not a perfect system. He also notes that hiding behind “the need for peer review” as an excuse for not discussing controversial results in public, as in the case of the NASA/phosphorus/arsenic news release and story, is not appropriate. 

Having once studied and worked in the field of scientific and electronic publishing, I am somewhat familiar with the system and the criticisms that Eisen encapsulates so well. Some of his criticisms can be attributed to the fact that humans are flawed. Operating any complex knowledge transfer system at a distance using volunteers with varying levels of familiarity with the topic being written about, and who are limited to interpreting a brief paper and minimal data through the lens of their own experience, will be bound to reach difficult-to-replicate conclusions that reflect a variety of personal biases. Bad stuff will occasionally get through and good stuff will occasionally not get published. Controversy will continue to swirl around a complex editorial process that so many have come to rely upon for professional validation and advancement.

Eisen’s comments, however, are not just a general critique of journal peer review. They are made in the context of the aforementioned controversial research finding that were first announced at a NASA press conference in 2010. The author of that study apparently has stated a preference that criticism of the study be made via the peer review process not in the open press.

While such a preference given the initial public release may be inappropriate I am also sympathetic with the author’s view that a completely open public forum may not be the most efficient format for discussions of complex and controversial research findings such as she first announced.

My view is that scientists should have an opportunity to talk “amongst themselves” to thrash out the pros and cons of complex and controversial topics. Whether or not that forum should necessarily be “peer reviewed” is the question. Given the various collaborative media now available to exchange, evaluate, and discuss information, a traditional refereed journal is not necessarily the best forum for such an interactive process. Nor should discussions of complex and potentially controversial topics be conducted “behind closed doors” — or behind pay walls.

In the case of the aforementioned research, which was at least partially funded by U.S. taxpayers, hiding information being exchanged by qualified researchers “until it is ready to be made public” is an old fashioned concept. Doing so would not only shortchange the public but would also promote an old fashioned view of science as a static process built on an edifice of facts created and maintained by ivory towered academics. 

When the need arises to evaluate and criticize new or controversial findings, perhaps we need a system that:

1. Allows scientists with relevant expertise to exchange information quickly and openly — and frankly — with the author.
2. Takes into account that the community of researchers addressing the topic may need to change over time since the topic of discussion may itself evolve.
3. Provides a mechanism for public access not only to the original content of the discussions but also to information about the participants themselves and their own qualifications.
4. Makes available “translations” of what is being discussed so that the interested public (and teachers, students, and law- and poicy-makers) have an opportunity to understand and appreciate the significance of what is being discussed.

Does such a system currently exist? If not, should it?

If it should exist, who should operate such a system and how would it be funded and managed?

Copyright (c) 2012 by Dennis D. McDonald