A protest on behalf of something as nebulous as “science” could mean anything, really. But when thousands of men, women, and children arrived in Washington, D.C., last month for a “March for Science,” their prime concern appeared to be that objective, scientific facts were losing ground in government. “What do we want? Evidence-based science!” protestors shouted in D.C. and around the world. “When do we want it? After peer review!”
This concern feels even more pressing now, in light of recent moves from the Trump administration to sever ties with independent scientists and even redefine what it means to be an “expert” in a given scientific field. The government’s scientific advisory boards, comprising academic scientists and other subject-area experts, have for decades been assigned the task of evaluating research data and its relationship to making regulations. But their work is being challenged from the top: On May 5, Environmental Protection Agency administrator Scott Pruitt fired nine members of a prominent review board and announced they might be replaced by industry scientists. The same day, the Department of the Interior announced that the work of a slew of advisory boards would be frozen, as Secretary Ryan Zinke was “reviewing [their] charter and charge.”
Pro-science activists are predictably enraged. Could the institution of peer review—a cornerstone of the scientific process and the hallmark of scientific rigor—really be dismantled in this way? But the administration’s opponents should be careful with their calls for greater scrutiny of data. For the last 25 years, conservatives have tried to interfere with regulation by attaching its approval to ever-more-complicated machineries of validation. The GOP and business interests tend to want more peer review, not less of it. Yet in the last few months, their self-interested position has been taken up with unlikely and unwitting glee by the Trump-hating masses.
Independent scientists have long had a role in government, supplying technical advice on subtle issues that relate to U.S. policy. The newly formed EPA began the practice of using peer review to design evidence-based regulations in the 1970s, utilizing a scientific advisory board to assess potential rules concerning, say, the banning or restriction of pesticides. By the early 1990s, the value of this system—at least as it had been implemented at that point—was being challenged. A four-month audit ordered by George H.W. Bush’s EPA administrator and published in 1992 found the agency’s science to be “of uneven quality,” while its “policies and regulations are frequently perceived as lacking a strong scientific foundation.” The report recommended a more standardized approach to scientific expertise, to “ensure a minimum level of quality assurance and peer review for all the science developed in support of Agency decision-making.”
The call for better peer review had, even at this early stage, taken on a right-wing tilt. As public-health scholars Roni Neff and Lynn Goldman explain, critics said the EPA and other agencies were sometimes sloppy in their application of the latest research and in ways that favored ill-considered, overreaching policies. GOP activists in Washington started pushing “sound science,” a movement that was purported to instill government decision-making with added rigor, integrity, and openness.
After their sweep to power in the 1994 midterm elections, Republican lawmakers rushed to implement House Speaker Newt Gingrich’s promised program of reform, including a bill to mandate “independent and external peer reviews” and cost-benefit analyses for regulations related to health, safety, or the environment. “On their face, these are worthy ideas,” the New York Times editorial board declared in February 1995. “But Mr. Gingrich’s approach … converts these useful concepts into a recipe for paralysis.” The Times editorial noted that, under the proposed law, agencies would’ve had to “endure a cumbersome 23-step review consisting of layers of ‘expert’ panels, some of which would include individuals or companies with a stake in the outcome.” In other words, stringent peer review would lead to fewer science-based interventions, not more of them.
The same rhetoric and strategy—to peer-review away the case for regulatory action— could in theory be applied across the board. Senate Republicans who worried the Endangered Species Act had led to “overzealous regulations based on emotion, not science,” for example, proposed a bill requiring rigorous peer review before any animal could be protected. But in the face of steady opposition from the Clinton White House, these efforts to spread peer review mostly came to nothing. While the Gingrich bill had support from several mainstream academic organizations, it also failed to pass.
It was only in the early 2000s, under George W. Bush, that “sound science” advocates made some minor progress on their platform. Scholars David Guston (a Slate contributor) and Stuart Shapiro explained in a 2006 paper how the effort to broaden and regularize peer review was taken up by the executive branch and implemented by Bush’s Office of Management and Budget. The administration tried to establish a set of federal standards for “when peer review is required and, if required, what type of peer review processes are appropriate.” All peer review should strive to ensure the “expertise and balance” of its panel members, the OMB insisted.
According to Shapiro, who used to work at OMB, there’s no evidence that the Bush-era guidelines were ever substantively enforced. “I don’t know that anything has changed,” he said. But in theory, the reforms were meant to broaden and regularize the process of peer review and ensure that it wouldn’t lean too far away from business interests. Although federal agencies had long tried to find “independent” peer reviewers, free of any financial ties to industry, Republican reformers felt they’d at times ignored the fact that academic scientists might have ties—financial or otherwise—to the agencies themselves.
The point of peer review had always been to draw on independent expertise, and academic scholars seemed best suited to this role, since they were likely to possess the subject knowledge necessary to evaluate highly technical questions and without any clear conflicts of interest. But now conservatives were saying that this definition of a “conflict of interest” was too narrow. Academic scientists could be compromised by funding from the same agencies for which they served on advisory boards. They could be friendly with the regulators. They could be politically inclined to favor government action. Meanwhile, any list of the most knowledgeable experts in any given field would reasonably include those who worked in a related business. Was it fair for only those scientists to be excluded? The OMB instructions said peer review procedures should consider, among other things, whether government-funded scientists “have sufficient independence from the federal agencies that support their work.”
The same argument is made, perhaps with even greater force, today. In February, House Science Committee Chairman Lamar Smith argued that scientific peer review should include more researchers from industry. “The EPA routinely stacks this board with friendly scientists who receive millions of dollars in grants from the federal government. The conflict of interest here is clear,” he said. “Simple changes, such as eliminating conflicts of interest, adding more balanced perspectives and being more transparent can go a long way to restoring the Agency’s credibility.”
That argument isn’t crazy. It’s certainly true that the makeup of a peer-review panel, and all the inclinations that its members bring, will help determine its decisions. It’s not like peer review converges, by its nature, on some objective answer to a scientific question; rather, it attempts to find a sensible, subjective synthesis of uncertain data points. And while we’ve seen ample and disturbing evidence of industry-enabled bias, it’s also true that academic, independent, progressive-minded researchers show a tendency, from time to time, to cherry-pick data that fits their predilections. One study of this phenomenon, by industry-funded scientists Mark Cope and David Allison, identified clear signs of what they called “white-hat bias” in the research literature on nutrition, including the misrepresentation of published findings and the selective reporting of results.
In determining the composition of a peer-review board, tough decisions must be made: How do you balance each scientist’s knowledge and experience against her commitment to a particular hypothesis? Institutions may be more forgiving of some forms of conflict than others. A recent Reuters investigation, for example, looked at how peer review is done at the International Agency for Research on Cancer, a group within the World Health Organization. According to the report, between 2012 and 2015 at least 1 in every 5 scientists involved in IARC’s “expert working groups” on potential carcinogens had been tasked with considering their own research. (It’s possible that many more were looking at the work of close colleagues.)
There are no simple, universal algorithms for making peer review effective. When we set the rules for membership on scientific advisory boards and suggest that one form of conflict might be more or less important than another, we’re instantiating values. How careful should the process be? How open to the latest science? How sensitive to uncertainty? The form of peer review that’s favored by the GOP sits at one extreme of this continuum: It would have the boards apply the strictest scrutiny to every claim and make consensus a hard-earned prize.
But however these decisions are made, peer review will always be an inherently conservative process. At its core, it serves to moderate the thrill for new ideas and fetter decision-making with long debates. That’s why it’s been so cherished by the right. Even if advisory boards are clogged with academics—and even if those academics are all rubber-stamping their friends’ research—the process of peer review will by its very nature slow down regulation.
Here’s just one example: In 2015, the Obama administration’s EPA moved to ban the widely used insecticide chlorpyrifos on account of its potential risk to children. The agency had been considering the evidence for such a ban for almost a decade. Scientific advisory panels have now weighed in on the potential harm three times. When the most recent round of peer review finished up last fall, its conclusions were still equivocal: There was reason to suspect the chemical might be dangerous at low exposures, the panel said, but “many uncertainties cannot be clarified” and certain more specific claims could not be supported by the data.
When the EPA decided late last year to move ahead with the ban in spite of this confusing assessment, Dow Chemical—the insecticide’s manufacturer—denounced its “fundamental scientific unsoundness.” Now, unsurprisingly, the Trump administration has taken Dow’s side. In March, the EPA reversed its earlier decision, citing in particular the peer-review reports from the scientific advisory panel. Those reports “have rendered numerous recommendations for additional study and sometimes conflicting advice for how EPA should consider (or not consider) the epidemiological data,” the agency said. In his own statement on the shift in policy, Pruitt claimed the administration was “returning to using sound science in decision-making—rather than predetermined results.”
Given the political valence of peer review, it’s peculiar that it’s become a cause célèbre for the March for Science. The “When do we want it? After peer review!” chant starts to make a bit more sense when you consider where it came from. As far as I can tell, the very first “When do we want it? After peer review!” sign was drawn up by a scruffy young gentleman in Washington, D.C., who carried the placard at Jon Stewart’s October 2010 Rally to Restore Sanity. Several media outlets singled out the message as among the event’s funniest, and the meme appears to have taken off from there.
What does that tell us? The Rally to Restore Sanity took place during neither the war-on-science reign of George W. Bush nor the anti-science administration of Donald Trump but rather in between, when respect for scientists had been momentarily restored. Stewart and his followers were not worried by the thought of regulators getting hamstrung by excessive layers of review. If anything, they had the opposite concern—that extremists on both sides were too emotional, too invested in pushing their agendas through without due consideration. What do we want, they asked? Sober, sane, evidence-based policies. And when do we want them? Only after lots of careful, reasoned conversations.
But Stewart’s call for compromise and moderation now seems outdated. Back in 2010, with a progressive in the White House, it felt appropriate to call for more deliberation in government. But times have changed. The people making the rules right now are looking for excuses not to act. Calls for peer review will only help them.
