A leading biologist at Harvard, Pardis Sabeti, has called out the replication movement in psychology, calling it a “cautionary tale” of how efforts to reform research may “end up destroying new ideas before they are fully explored.” Her argument, in short, is that the “vicious” debate over statistical errors in that field has only stymied further progress. There’s “a better way forward,” Sabeti says, “through evolution, not revolution.” For comparison, she describes what happened in her own field of human genomics: A rash of false-positive results gave way about 10 years ago, without much fuss or incivility, to a new and better way of doing science. “We emerged more engaged, productive, successful, and united,” she says. Now it’s time for psychologists to put aside their pettiness and try to do the same.
Sabeti’s call to end the revolution, which appeared in Sunday’s Boston Globe, has been ballyhooed by several of her well-known campus colleagues. “Put a lid on the aggression & call off the social media hate mobs,” wrote Steven Pinker on Twitter. Sabeti “has written one of the smartest essays about the politics of social psychology that I’ve ever read,” said Daniel Gilbert. “Compelling piece … on how 2 scientific fields made major course corrections,” said Atul Gawande.
These kudos are misguided. While Sabeti makes it sound as if the reformers in psychology all behave like bullies, that’s far from the truth. She also suggests changes in genomics were implemented without rancor, as if rival scientists came together to sing Kumbaya. That’s not true, either.
Sabeti’s depiction of debates within psychology is particularly egregious. According to her piece, the field’s “revolutionaries” compose a nasty coterie of “scientists and Internet bloggers” who are bent on tearing others down. Sabeti names seven victims of this bullying, but it’s clear the main victim she has in mind is Amy Cuddy, the Harvard Business School professor who rose to fame on the basis of her research into “power posing.” In Cuddy’s viral TED talk on the topic, she claimed that holding certain postures can change your hormone levels and behavior, and help to elevate those “with no resources and no technology and no status and no power.” This turned out to be untrue: Numerous, rigorous attempts to replicate her findings have shown no effect from power posing when it comes to boosting hormones, and scant evidence of changes to behavior.
Sabeti seems largely unfamiliar with this literature. Her essay claims (as Cuddy does) that power posing has a real effect—that it makes people feel more powerful than they did before. But this result isn’t that surprising, and it wasn’t even treated as a main result in Cuddy’s own, original power-posing work. Furthermore, even these effects on “felt power” are somewhat small, on average, and could well be linked to subjects’ expectations. (For one thing, studies find better evidence of a power boost for people who are already familiar with Cuddy’s TED talk than for those who aren’t.)
According to Sabeti, we’d know more about the science of power posing if the replicators’ nastiness hadn’t stifled meaningful research. “The criticisms of Cuddy and of other researchers swept up in the revolution have been so rife with vitriol,” she claims, “that it has left researchers wary of publishing new findings.” (That’s a bold, unfounded claim: To my knowledge, scientists rarely skip a chance to pad their résumés with published papers.) Sabeti continues: When scientists such as Cuddy try to defend themselves with evidence, they’re “shouted down,” and their ideas are “dismissed.” “Bullies pile on and bystanders fearfully turn a blind eye,” she says, adding that these disagreements often play out in the dangerous domain of social media, where “there is an insidious temptation to mistake being critical for being right.”
The Globe piece cites 10 examples of this cruel and hurtful rhetoric—10 places online where researchers have been attacked for being “defensive,” “dishonest,” “sloppy,” “incompetent,” “in deep denial,” “mockworthy,” and “anti-replication,” or accused of purveying “pseudoscience,” “junk science,” and “tabloid fodder.” “The attacks on these scientists have become so personal and so threatening that there may be no one in the field willing to speak up on their behalf,” Sabeti warns.
Who, exactly, are the marauding revolutionaries behind the attacks above? Follow the links on the Globe’s website and you’ll find that seven of Sabeti’s 10 examples come from a single source: Columbia statistician Andrew Gelman. Six are from his blog; the seventh appeared in an article he co-authored with Kaiser Fung for Slate. Two more examples are drawn from anonymous commenters on Gelman’s blog, and the last is pulled from an article by FiveThirtyEight’s (scrupulous, award-winning) science journalist Christie Aschwanden.
In other words, when Sabeti talks about a suffocating atmosphere of terror in social psychology, she’s referring almost entirely to critiques from one professor, who happens not to be a member of that field, along with some readers of his blog. And while she names seven victims of these attacks—Cuddy, Angela Duckworth, Kristina Durante, Barbara Fredrickson, Simone Schnall, Roy Baumeister, and Fritz Strack—almost all the attacks she cites are about Cuddy.
It’s true critiques of Cuddy’s work—and critiques of those critiques—have gotten lots of press. (That’s due to power posing’s public profile, stemming from Cuddy’s TED talk.) But these critiques represent a tiny fraction of the movement to reform the field of social psychology. The efforts to promote more open and transparent research methods, and to question findings from the past, are both comprehensive and forgiving, as a rule. Not every replication failure ends up in a fight, and most discussions of these issues have been forward-looking. Indeed, social psychology has already done far more to solve its problems—implementing systems at journals and in labs that help prevent the spread of false-positive results—than most other scientific fields.
Yet Sabeti stays with Cuddy’s story, as opposed to any other. But even there, I think she’s off the mark. In the kicker to her piece, she instructs psychology to “manage its revolution” by “consciously promoting both civility and rigor.” This new attitude would permit scientists, she says, to conduct meaningful investigations that “shed light on psychological phenomena” like power posing. “Some hypotheses, such as the idea that specific poses temporarily alter hormone levels in a person’s body, may or may not bear out,” she says—but we’ll never know until the revolution has been put down, and scientists feel safe enough to pursue the truth.
That’s exactly backward, though. The claim that power posing changes hormone levels and behavior hasn’t just been studied; one could say it’s been grossly overstudied. Countless hours have been spent compiling data from rigorous, preregistered power-pose experiments that were launched in response to Cuddy and her colleagues’ original, shaky findings. Last summer, the journal Comprehensive Results in Social Psychology published a special issue devoted to more than half a dozen such efforts, all conducted with a maximum of civility and rigor. One of Cuddy’s own co-authors on the original power-posing research, Dana Carney of the University of California, Berkeley, helped to oversee the project. “Part of our aim … was to show how researchers could coordinate and cooperate in an effective and efficient manner,” Carney and her colleagues wrote in the special issue’s introduction. “It is obvious that the researchers contributing to this special issue framed their research as a productive and generative enterprise,” they added, “not one designed to destroy or undermine past research.”
Carney and the others did exactly what Sabeti suggests: They organized a collaborative and diplomatic investigation of the power-pose phenomenon. And what came out of this massive, costly enterprise? From the special issue’s summary: “There was virtually zero effect of power poses on any of the behavioral or hormonal measures.”
Sabeti says nothing whatsoever about this careful and constructive work. Instead, she hints that if power posing’s critics aren’t bullies, they must have been brainwashed by the replication jerks. It was a fear of “vicious reproach,” she says, that led Carney to “disavow [her] own work in sweeping claims.” Never mind the alternative (and more respectful) explanation: that Carney applied her scientific judgment to the evidence at hand and went where the data led her.
If Sabeti paints a foul and distorted picture of the replication movement in psychology, what about the claims she makes about her own field of research? “With advances in genomics and better corrections for testing multiple ideas, it became clear that some of the results they previously found were just statistical flukes,” she writes. While the bullies in psychology allegedly reacted to such problems with vituperative assaults, she claims the genomics crowd took a different path: “By and large, leaders in my field consciously chose to evolve through greater rigor and collaboration rather than by ‘gotcha.’ We engaged other geneticists as if they came from an honest place. We returned to an agnostic baseline. … The field continually corrected itself.”
This happy story elides some important details. In the early days of human genome research, up until the mid-2000s, scientists did the best they could with whatever technology they had.
In practice, that meant looking for links between various diseases and specific genes that might be relevant; for example, they might study whether people with different versions of a gene related to alcohol metabolism might have different propensities for drinking to excess. But as their tools improved, and their data sets grew many times more massive, they were no longer constrained by the guesswork of the old “candidate gene” approach. Instead, they could test very large groups of people, scanning their entire genomes all at once, to find potential links between genetic makeups and disease.
In other words, the “different path” that genomics has taken in the past decade may have had less to do with the primacy of civil discourse than with the changing nature of the field itself.
All of a sudden, data was cheap and far more plentiful. This made a huge difference with respect to which research questions could be asked, and how they might be answered.
As the field has evolved, genomics has had its share of testy moments. In 2009, for example, a team led by Neil Risch published a meta-analysis in the Journal of the American Medical Association that debunked one of the field’s most prominent findings from the earlier, candidate-gene literature: that people who possess a certain variant of a serotonin transporter gene are more likely to sink into deep depressions after experiencing stressful life events. The Risch paper blew up this hypothesis, along with lots of others like it—what the paper’s authors called “other nonreplicated genetic associations.” Risch et al. also hinted that massive grants for genomic research should be redirected—away from the old candidate-gene studies and toward big data research like their own.
This all happened in the early days of social media. While it may be true that geneticists weren’t slagging colleagues on their Myspace pages, they did express their vitriol in other settings.
“That Risch paper is bullshit,” one anonymous scientist told writer David Dobbs for a Wired piece published in 2012. Others who critiqued the old approach were allegedly shouted down in public for being “unscientific.” And yes, in more recent years, some genomics fights have broken out on Twitter.
In her essay for the Globe, Sabeti seems at first to speak to the reformers in psychology, urging them to act with more composure and respect. But the piece’s headline hints at something else. “For better science, call off the revolutionaries,” it says, as if a “calling off” could be ordered from the top, decreed by some generalissimo psychologist. Perhaps that might be possible if this were an issue localized to a single subfield of research, or if it were about one specific feud between a pair of academics. (Maybe Andrew Gelman will get tired of blogging about Amy Cuddy.) But the current revolution in psychology, like the one that happened in genomics, isn’t under anyone’s command. And while many scientists have done their best to keep things diplomatic, both fields are already strewn with defunct ideas—fallen bodies of literature, and career-defining work that’s now been left for dead. It’s grim to watch a theory get dismantled, even when that theory happens to be wrong. But that’s how science works.
