You Think Your Science Is Correct? Prove It

The state of the universe.
Aug. 14 2012 10:46 AM

Good Scientist! You Get a Badge.

Precious research money is wasted on unreal results, but we can change the culture of science.

(Continued from Page 1)

Why would you do this? For one thing, you’ll get a second paper out of the experience, and scientists are judged in part by the number of papers on their CV. Scientists often find it hard to publish replication studies. Iorns had to send her SATB1 paper to a number of journals before getting it published, despite the fact that it revealed that investigating SATB1 for a cancer cure would be a waste of time. The journal PLoS ONE has agreed to publish any study that comes out of the Reproducibility Initiative.

A number of other journals have also agreed to add a badge to all papers that have been replicated through the Reproducibility Initiative.* Think of it as a scientific Good Housekeeping seal of approval. It shows that you care enough about the science—and are confident enough in your own research—to have someone else see if it holds up.

If the project takes off, it’s possible that funding agencies would require a validation test. That way they’d avoid wasting grant money on following up on findings that don’t withstand an independent test. The badge could also become important for commercial development of scientific research. If you want to license your cancer research to a pharmaceutical company, you may have to show your badge as evidence that the investment won’t evaporate.

Advertisement

Iorns has gotten some big names on board as advisers, such as John Ioannidis of Stanford University, author of the game-changing 2005 paper, “Why Most Published Research Findings Are False.” It will be fascinating to see how Iorns’ team fares. They mostly come from the world of biomedical research, and it’s easiest to see how their strategy could work in that community. Iorns thinks that her strategy could work outside that world as well. The arsenic life researchers could have hired an outside firm to culture the bacteria and sequence its DNA, for example.

The Reproducibility Initiative will probably fare best when the science involved can be easily duplicated with standard pieces of technology. The more cutting edge the research, the fewer people will be able to replicate it. In June, Jay Shendure of the University of Washington and his colleagues sequenced the entire genome of an 18-week fetus based only on blood from the mother and saliva from the father. It’s a tour de force of DNA isolation and computer analysis—one that only a few labs in the world today could manage.

Other scientific fields could use replication studies, too. Experimental psychology—which grabs lots of headlines about the secret workings of the mind—is in particularly dire need. As journalist Ed Yong has reported, psychologists often goose up the statistical significance of their research to get it published, while failed replication studies regularly get rejected by psychology journals. In January some psychologists tried to reverse this trend by setting up a website called PsychFileDrawer, where researchers can post unpublished attempts to replicate psychology studies. So far they’ve attracted only 13 entries.

At the moment, the Reproducibility Initiative can’t help the mess in psychology. There are no psychologists at the Science Exchange ready to offer their services. But perhaps that, too, might change. Maybe the Reproducibility Initiative—and efforts like it—can transcend these logistical limits. Iorns and her colleagues are trying to reprogram the incentives in science. Right now, a lot of the incentives to take extra care rather than rushing to publish research are on the stick side of the carrot and stick equations—first and foremost, the fear that your paper gets retracted.

“If you are retracted, it’s career-breaking, and you’re a fraudulent scientist. It’s very negative,” Iorns says. “We said, ‘Why don’t we reward scientists who use high-quality data?’ Eventually the culture shifts from just funding originality, and instead we shift to rewarding things that are really true.”

Update, Dec. 28, 2012: The authors of the original Nature study on SATB1 have disputed Iorns’ conclusion, and some subsequent studies by other scientists have supported the original link to breast cancer. Others have not.

Correction, Aug. 14, 2012: This article originally stated that Nature has agreed to add a badge to papers that have been replicated through the Reproducibility Initiative. Nature is planning to link its papers to relevant replication studies published in the journal PLoS ONE. (Return to the corrected sentence.)

TODAY IN SLATE

Politics

The Democrats’ War at Home

How can the president’s party defend itself from the president’s foreign policy blunders?

An Iranian Woman Was Sentenced to Death for Killing Her Alleged Rapist. Can Activists Save Her?

Piper Kerman on Why She Dressed Like a Hitchcock Heroine for Her Prison Sentencing

Windows 8 Was So Bad That Microsoft Will Skip Straight to Windows 10

Homeland Is Good Again! For Now.

Politics

Cringing. Ducking. Mumbling.

How GOP candidates react whenever someone brings up reproductive rights or gay marriage.

Music

How Even an Old Hipster Can Age Gracefully

On their new albums, Leonard Cohen, Robert Plant, and Loudon Wainwright III show three ways.

The U.S. Has a New Problem in Syria: The Moderate Rebels Feel Like We’ve Betrayed Them

We Need to Talk: A Terrible Name for a Good Sports Show by and About Women

Trending News Channel
Oct. 1 2014 1:25 PM Japanese Cheerleader Robots Balance and Roll Around on Balls
  News & Politics
The World
Oct. 1 2014 12:20 PM Don’t Expect Hong Kong’s Protests to Spread to the Mainland
  Business
Moneybox
Oct. 1 2014 2:16 PM Wall Street Tackles Chat Services, Shies Away From Diversity Issues 
  Life
The Eye
Oct. 1 2014 1:04 PM An Architectural Crusade Against the Tyranny of Straight Lines
  Double X
The XX Factor
Oct. 1 2014 2:08 PM We Need to Talk: Terrible Name, Good Show
  Slate Plus
Behind the Scenes
Oct. 1 2014 3:24 PM Revelry (and Business) at Mohonk Photos and highlights from Slate’s annual retreat.
  Arts
Brow Beat
Oct. 1 2014 3:02 PM The Best Show of the Summer Is Getting a Second Season
  Technology
Future Tense
Oct. 1 2014 3:01 PM Netizen Report: Hong Kong Protests Trigger Surveillance and Social Media Censorship
  Health & Science
Science
Oct. 1 2014 2:36 PM Climate Science Is Settled Enough The Wall Street Journal’s fresh face of climate inaction.
  Sports
Sports Nut
Sept. 30 2014 5:54 PM Goodbye, Tough Guy It’s time for Michigan to fire its toughness-obsessed coach, Brady Hoke.