The case against peer review.
More sweeping changes have also been suggested. Well-trained, full-time editors and professional statisticians might be able to perform the functions of peer review on their own. Or scientists en masse might be recruited: Paul Ginsparg, who runs a digital archive for unpublished physics papers, has suggested that putting "preprints" of scientific papers on the Web could let the community as a whole decide which papers are most useful. Unpublished work could be tracked by an objective measure—like how often it's cited or downloaded—and then passed along for formal publication. Government funders like the NIH could hire professional reviewers to evaluate grants, or they could replace grants with cash prizes for successful research.
When journal editors are asked about these ideas, they often quote Winston Churchill's line, "Democracy is the worst form of government except all those other forms that have been tried from time to time." Or rather, they quote other journal editors quoting that line. But it's a poor analogy, since few alternatives to peer review have been tried in modern times. And democracy isn't really a good description of peer review, either. Sure, peer review allows scientists to participate in a system of self-governance. But wouldn't BMJ's policy of open review or Ginsparg's proposal for Web-published preprints be far more democratic?
So far, though, the Churchill quoters are winning. A good study of peer review will take more than a small band of outsiders working part time with little money. We need a coordinated effort by scientific journals, funding agencies, and research scientists. It wouldn't be that expensive. But the funds have not been forthcoming. Instead, the federal government recently proposed using peer review "to improve the quality, objectivity, utility, and integrity of information" that it gives to the publicfor things like FDA drug requirements and HHS dietary guidelines. Ironically, most scientists hated the idea when it was first presented; the original version of the proposal stated that only researchers from private industry, who don't receive federal funding, would get to participate. (That provision has since been eliminated. *) But forget arguing over the details of peer review. Let's first figure out whether it works.
Correc tion, April 8, 2005: The original version of this column missated a provision of the federal government's proposal on peer review. The revised version of the proposal would allow federally funded scientists to participate.