The Dangers of Data

A blog about business and economics.
Aug. 7 2012 5:38 PM

The Dangers of Data

Betsey Stevenson and Justin Wolfers hail the way increases in computing power are opening vast new horizons of empirical economics.

I have no doubt that this is, on the whole, change for the better. But I do worry sometimes that social sciences are becoming an arena in which number crunching sometimes trumps sound analysis. Given a nice big dataset and a good computer, you can come up with any number of correlations that hold up at a 95 percent confidence interval, about 1 in 20 of which will be completely spurious. But those spurious ones might be the most interesting findings in the batch, so you end up publishing them!

Advertisement

A bit more subtly, there's also the problem of Milton Friedman's thermostat. Take a room with a furnace that's regulated by a really good thermostat. Your data is going to show that the amount of fuel burned by the furnace is uncorrelated with the temperature in the room. Thus you'll discover that burning fossil fuels doesn't cause heat. Oooops! In the natural sciences what you'd do with that finding is run some experiments, and you'd figure out what was really happening. But it's often difficult (or simply unethical or inhumane) to run proper social-science experiments. A country with a really sharp central bank ought to be like a room with a really good thermostat—variations in economic conditions will be statistically driven by real shocks, which can lead to the misleading conclusion that central bank policy doesn't matter. And the best way to really test what central banks can and can't do would be to run some experiments in which they deliberately do crazy stuff, but that's not going to happen.

All of which is to say that pure number crunching can be a dangerous business. When it's hard to run the math, empirical inquiry ends up constrained by hypotheses that are plausible. When it's possible to run experiments, we can dive deep into our data and try to really pin the issue down. But a large dataset and a powerful computer, untempered by theory or experimentation, can create a lot of room for mischief.

Matthew Yglesias is the executive editor of Vox and author of The Rent Is Too Damn High.

TODAY IN SLATE

Politics

The Democrats’ War at Home

How can the president’s party defend itself from the president’s foreign policy blunders?

Congress’ Public Shaming of the Secret Service Was Political Grandstanding at Its Best

Michigan’s Tradition of Football “Toughness” Needs to Go—Starting With Coach Hoke

A Plentiful, Renewable Resource That America Keeps Overlooking

Animal manure.

Windows 8 Was So Bad That Microsoft Will Skip Straight to Windows 10

Politics

Cringing. Ducking. Mumbling.

How GOP candidates react whenever someone brings up reproductive rights or gay marriage.

Building a Better Workplace

You Deserve a Pre-cation

The smartest job perk you’ve never heard of.

Hasbro Is Cracking Down on Scrabble Players Who Turn Its Official Word List Into Popular Apps

Florida State’s New President Is Underqualified and Mistrusted. He Just Might Save the University.

  News & Politics
Politics
Sept. 30 2014 9:33 PM Political Theater With a Purpose Darrell Issa’s public shaming of the head of the Secret Service was congressional grandstanding at its best.
  Business
Moneybox
Sept. 30 2014 7:02 PM At Long Last, eBay Sets PayPal Free
  Life
Gaming
Sept. 30 2014 7:35 PM Who Owns Scrabble’s Word List? Hasbro says the list of playable words belongs to the company. Players beg to differ.
  Double X
The XX Factor
Sept. 30 2014 12:34 PM Parents, Get Your Teenage Daughters the IUD
  Slate Plus
Behind the Scenes
Sept. 30 2014 3:21 PM Meet Jordan Weissmann Five questions with Slate’s senior business and economics correspondent.
  Arts
Brow Beat
Sept. 30 2014 8:54 PM Bette Davis Talks Gender Roles in a Delightful, Animated Interview From 1963
  Technology
Future Tense
Sept. 30 2014 7:00 PM There’s Going to Be a Live-Action Tetris Movie for Some Reason
  Health & Science
Medical Examiner
Sept. 30 2014 11:51 PM Should You Freeze Your Eggs? An egg freezing party is not a great place to find answers to this or other questions.
  Sports
Sports Nut
Sept. 30 2014 5:54 PM Goodbye, Tough Guy It’s time for Michigan to fire its toughness-obsessed coach, Brady Hoke.