A year ago, researchers from the University of Virginia published the findings from their attempt to repeat 100 published psychological experiments to see whether they could get the same results. The conclusion of the study indicated that only one-third to one-half of the original findings yielded similar outcomes. This study, known as the Reproducibility Project, brought attention to the fact that a great deal of scientific research produced today is not reproducible. Even the Reproducibility Project’s results were challenged when put to the test by other researchers. The ability to reaffirm findings is an important part of the scientific method. Yet the reality is that billions of dollars fund studies whose results don’t hold up when put under the microscope again.
On April 21, Future Tense—a partnership of Slate, New America, and Arizona State University—hosted an event in Washington, D.C., to discuss this crisis in biomedical research. The two conversations featured scientists, researchers, and journalists who are addressing the problem in diverse ways. It quickly became clear that this crisis is not isolated to any one field of research—and despite increased attention to the issue this past year, it’s not new.
Lawrence Tabak, principal deputy director of the National Institutes of Health, noted that he first became aware of this problem more than 35 years ago as an assistant professor working on a major grant application. After failed attempts to replicate a key study done by a prominent scientist, Tabak realized the dilemma he encountered went beyond the replication of data and results; rather, it hinted at a larger cultural problem in the scientific enterprise. According to Tabak, perverse incentives continue to allow, if not encourage, scientists to get away with publishing invalid research. In the publish-or-perish world of research institutions, professional development often depends on a scientist’s ability to produce exciting results at a rapid pace. Furthermore, Brian Nosek, executive director and co-founder of the Center for Open Science, which coordinated the Reproducibility Project, pointed out that reproducing studies will create transparency and higher scrutiny of the scientific process, which could encourage researchers to value quality of publication, rather than quantity.
It’s difficult to change one’s approach to the scientific process and incentives midcareer—which is why it’s important to think about students in these conversations. Emma Frow, assistant professor in the School of Biological and Health Systems Engineering and the School for the Future of Innovation in Society at Arizona State, believes that science education can instill new values in future generations who are not yet under the pressure to publish research. The goal is to create a community-oriented culture and an incentive structure that emphasizes process, not prestige.
But the current reproducibility problem isn’t just about researchers scrambling to get ahead; other variables can affect the reliability of results. For instance, Carolyn Compton, professor of life science at ASU and former pathologist, discussed how the specimens used in labs are often unregulated leaving their consistency and quality unreliable. According to Compton, we often know more about the beef in our supermarkets—thanks to regulation by the Food and Drug Administration—than we know about the samples that are at the heart of bioresearch. In studies that require precise measurements and astute attention to detail, this kind of inconsistency not only influences the results of the study but its ability to be reproduced.
Yet Arturo Casadevall, who is the Alfred and Jill Sommer professor and chair of the W. Harry Feinstone Department of Microbiology and Immunology at Johns Hopkins University, argued that uncertainties are just par for the course. Casadevall reminded the audience that “all science is provisional,” and as scientists continue to investigate and learn, their approaches change, as do their results. Therefore, we should not search for the “perfect” research studies as much as we should bring critical attention to all of them, with emphasis on method and standards. Richard Harris, visiting scholar at ASU’s Consortium for Science, Policy and Outcomes at ASU, and science correspondent on leave from National Public Radio to write a book on the crisis in reproducibility, encourages us to embrace doubt and understand that unknowns are part of the scientific process. The infrastructure surrounding scientific inquiry may be flawed but it’s our search for the truth that directs our ever evolving inquiry into the known and unknown.
You can watch the full event on New America’s website. Also in Slate:
- “The Unintended Consequences of Trying to Replicate Research,” by Ivan Oransky and Adam Marcus
- “The Reproducibility Crisis Is Good for Science,” by Monya Baker
- “Cancer Research Is Broken,” by Daniel Engber
