The Dangers of Data

A blog about business and economics.
Aug. 7 2012 5:38 PM

The Dangers of Data

Betsey Stevenson and Justin Wolfers hail the way increases in computing power are opening vast new horizons of empirical economics.

I have no doubt that this is, on the whole, change for the better. But I do worry sometimes that social sciences are becoming an arena in which number crunching sometimes trumps sound analysis. Given a nice big dataset and a good computer, you can come up with any number of correlations that hold up at a 95 percent confidence interval, about 1 in 20 of which will be completely spurious. But those spurious ones might be the most interesting findings in the batch, so you end up publishing them!

Advertisement

A bit more subtly, there's also the problem of Milton Friedman's thermostat. Take a room with a furnace that's regulated by a really good thermostat. Your data is going to show that the amount of fuel burned by the furnace is uncorrelated with the temperature in the room. Thus you'll discover that burning fossil fuels doesn't cause heat. Oooops! In the natural sciences what you'd do with that finding is run some experiments, and you'd figure out what was really happening. But it's often difficult (or simply unethical or inhumane) to run proper social-science experiments. A country with a really sharp central bank ought to be like a room with a really good thermostat—variations in economic conditions will be statistically driven by real shocks, which can lead to the misleading conclusion that central bank policy doesn't matter. And the best way to really test what central banks can and can't do would be to run some experiments in which they deliberately do crazy stuff, but that's not going to happen.

All of which is to say that pure number crunching can be a dangerous business. When it's hard to run the math, empirical inquiry ends up constrained by hypotheses that are plausible. When it's possible to run experiments, we can dive deep into our data and try to really pin the issue down. But a large dataset and a powerful computer, untempered by theory or experimentation, can create a lot of room for mischief.

Matthew Yglesias is the executive editor of Vox and author of The Rent Is Too Damn High.

TODAY IN SLATE

Politics

Blacks Don’t Have a Corporal Punishment Problem

Americans do. But when blacks exhibit the same behaviors as others, it becomes part of a greater black pathology. 

I Bought the Huge iPhone. I’m Already Thinking of Returning It.

Scotland Is Just the Beginning. Expect More Political Earthquakes in Europe.

Lifetime Didn’t Think the Steubenville Rape Case Was Dramatic Enough

So they added a little self-immolation.

Two Damn Good, Very Different Movies About Soldiers Returning From War

Medical Examiner

The Most Terrifying Thing About Ebola 

The disease threatens humanity by preying on humanity.

Students Aren’t Going to College Football Games as Much Anymore, and Schools Are Getting Worried

The Good Wife Is Cynical, Thrilling, and Grown-Up. It’s Also TV’s Best Drama.

  News & Politics
Weigel
Sept. 20 2014 11:13 AM -30-
  Business
Business Insider
Sept. 20 2014 6:30 AM The Man Making Bill Gates Richer
  Life
Quora
Sept. 20 2014 7:27 AM How Do Plants Grow Aboard the International Space Station?
  Double X
The XX Factor
Sept. 19 2014 4:58 PM Steubenville Gets the Lifetime Treatment (And a Cheerleader Erupts Into Flames)
  Slate Plus
Slate Picks
Sept. 19 2014 12:00 PM What Happened at Slate This Week? The Slatest editor tells us to read well-informed skepticism, media criticism, and more.
  Arts
Brow Beat
Sept. 19 2014 4:48 PM You Should Be Listening to Sbtrkt
  Technology
Future Tense
Sept. 19 2014 6:31 PM The One Big Problem With the Enormous New iPhone
  Health & Science
Bad Astronomy
Sept. 20 2014 7:00 AM The Shaggy Sun
  Sports
Sports Nut
Sept. 18 2014 11:42 AM Grandmaster Clash One of the most amazing feats in chess history just happened, and no one noticed.