The case against peer review.

Health and medicine explained.
April 5 2005 2:03 PM

Quality Control

The case against peer review.

1_123125_123104_2112652_2116243_2116248_050405_neubecker_peer

In September 2001, the Journal of Reproductive Medicine weighed in on the healing power of God. A Columbia University research group reported that patients at a fertility clinic in Seoul were twice as likely to get pregnant when Christians prayed for them. Within a month, the study was in the New York Times science section and on Good Morning America, where the medical editor for ABC News called it "very well done" and opined that "getting pregnant involves a lot of biological, psychological, maybe even spiritual factors that we don't yet understand."

Daniel Engber Daniel Engber

Daniel Engber is a columnist for Slate

The prayer study has since fallen from grace. Scientists around the world wrote angry letters to the journal attacking the methodology, and the research-protections office of the Department of Health and Human Services looked into whether the subjects had properly given consent. Last year, the study's senior author removed his name from the paper, saying that he hadn't directly participated in the research. The real lead author will not discuss the work, and the third author—a parapsychologist, lawyer, and convicted con man—is now serving time in a federal prison (for an unrelated charge of fraud).

Advertisement

Why did this quackery get so far before being exposed? The prayer study seemed legitimate because it appeared in the pages of a "peer-reviewed" medical journal. That means the paper was vetted by an independent panel of experts in the field.

Peer review is the gold standard of modern science. For medical researchers and other scientists, it's the gateway to funding, publication, and career advancement. When they apply for government grants from the National Institutes of Health or the National Science Foundation, their proposals are reviewed by a panel of their colleagues. When they submit their completed work for publication, journals and university presses ask for the opinions of others in the field. And when they apply for jobs or tenure, scientists are judged largely on the basis of their peer-reviewed publications.

Scientists give peer review so much authority because they view it as a part of the grand tradition of scientific inquiry—an extension, even, of the formal experimental method. Peer evaluation is the endpoint of a cautious progression from theories and predictions to experiments and results. The system dates from the 1700s, when the Royal Society of London set up a "Committee on Papers" with the power to solicit expert opinions. It became the standard for scientific publication only after World War II, when the dramatic expansion of scientific research swamped journal editors and made them look to outsiders for help. Ever since, scientists have claimed that peer review filters out lousy papers, faulty experiments, and irrelevant findings. They say it improves the quality of an accepted paper by providing helpful comments for revision. And they can't imagine a better way to accomplish these goals.

So, what explains the Columbia prayer study? Journal editors will tell you that peer review is not designed to detect fraud—clever misinformation will sail right through no matter how scrupulous the reviews. But the prayer study wasn't a clever fraud. It was sprinkled with suspect elements, not the least of which was a set of results that violated known laws of science. The authors also used a needlessly convoluted experimental design; these and other red flags in the study have been cataloged on the Web by obstetrician and enthusiastic debunker Bruce Flamm. Even on its own terms, then, as a filter for lousy papers and bad experiments, peer review of the Columbia prayer study was a spectacular failure. Here's the problem: Despite its authority and influence over every aspect of the scientific community, no one has ever shown that peer review accomplishes anything at all.

In 1986, a deputy editor at the Journal of the American Medical Association named Drummond Rennie announced the first scholarly conference on peer review. A series of high-profile cases of scientific fraud had hit the medical journals in the early '80s, and lousy papers were making their way into print on a regular basis. Editors like Rennie were in the mood for critical self-examination.

If peer review didn't work, it was an extraordinary waste of time. Rennie proposed to study whether the system, with all its shortcomings, actually improved the quality of published research. He didn't invent the idea—another medical editor, Steven Locke, had just published a book that asked the same question—but he nurtured and led the small group of maverick medical editors who were ready to challenge the status quo. In 1986, when Rennie announced the First International Congress on Peer Review and Biomedical Publication, there were only a handful of (peer-reviewed) papers on the topic; now there are several hundred and Rennie's congress convenes every four years. A cartoon in a special issue of JAMA devoted to the group's 2001 meeting in Barcelona shows the white-bearded Rennie dressed like Moses, leading his fellow scientists and editors through the desert of Sinai.

Rennie and his companions have spent almost 20 years in the desert, yet the golden calf is still intact. The study of peer review turns out to be tremendously difficult. To test whether it works, you'd need to compare the quality of papers that had gone through peer review with the quality of those that hadn't. But how would you get papers for the control group, given all the professional benefits that come with peer review? And assuming you could convince scientists to forgo the process, how could you objectively judge the quality of the papers? At Rennie's fifth congress this year in Chicago, several hundred studies will be presented, but no one will claim to have answered the big question: Does peer review work?

In the meantime, the system is threatening to collapse under its own weight (click here for more about how). What's the fix? Rennie's gang has produced a few tantalizing bits of data. One set of studies shows that revealing the identities of reviewers has no effect on the quality of their efforts, contradicting the standard argument that anonymity fosters rigorous criticism and high standards. In response to findings like these, some journals, like the British Medical Journal, have switched to "open review," telling authors who has reviewed their papers.

  Slate Plus
Slate Picks
Dec. 19 2014 4:15 PM What Happened at Slate This Week? Staff writer Lily Hay Newman shares what stories intrigued her at the magazine this week.