Why So Much Economics Data Is Garbage

Commentary about business and finance.
Jan. 7 2013 3:05 PM

Economists Use Shoddy Data

All too often, economists’ and policymakers’ theories are based on terrible statistics.

How can we get better data?
How can we get better data?


Economists are a smart bunch, and in San Diego over the weekend for the annual meeting of the American Economic Association lots of impressive analytical chops were on display. Sometimes, though, the analysis seems to badly overreach the data. A well-attended panel discussion about whether economics is hopelessly riven by ideological disagreement, for example, proceeded even though the IGM Chicago survey it’s based on isn’t even close to being a statistically valid survey. And the issue recurs even outside the realm of such “fun” discussions and infects much more sober research. No grand theory is going to be much better than the data it’s based on, and the underlying quality is often distressingly poor.

A worse-attended, dramatically lower-charisma discussion occurred the following day about a dully titled paper on “Cyclical Variation in Productivity Using the ATUS” by Michael Burda, Daniel Mamermesh, and Jay Stewart.

The point of the paper is that theories of the macroeconomy often depend crucially on ideas about the direction of productivity in the labor force. And to know how productive workers are on a per-hour basis, you need to know how many hours people work. This, in turn, is a messy business. The data traditionally come from two sources. One, the Current Employment Survey (CES), is a survey of employers and thus misses the self-employed and can’t tell how many people are working multiple jobs. For the vast majority of its history the CES also missed supervisory workers. The other, the Current Population Survey (CPS), asks individuals and is thus more comprehensive. But asking people what they did last week often generates inaccurate answers.

A better solution is to use the American Time Use Survey (ATUS), which asks people to keep a real-time diary, ensuring that numbers add up to 24 hours in a day and reducing memory error.

ATUS data has been around for a while, but as with the CES and CPS, its hours-worked series isn’t really used for anything despite being generally considered more accurate. What Burda, Hamermesh, and Stewart did was recalculate hourly productivity data with ATUS hours worked as the denominator. The scale of the change here is not enormous—the data series differ, but not all that much—but one interesting thing did happen. According to the standard data, productivity used to fall during recessions but has risen during recent recessions. According to the ATUS data, productivity still falls during recessions. The authors disclaim any ambitions to become macroeconomic theorists but did draw attention to a key point. The traditional view had been that during recessions employers engage in “labor hoarding,” hanging on to more workers than necessary in order to avoid wasteful firing and rehiring. More recently, hoarding seemed to have gone away. Firms instead took recessions as opportunities to trim the fat, firing low-quality workers and raising average productivity. This apparent switch in firm behavior has been thought by many to relate to the rise of “jobless recoveries.” Except now it looks like the change may never have happened.

So a small fussy point ends up undermining a big-picture argument. And it happens not just in academia, but in policy circles as well. The Obama administration, for example, took office in January of 2009 and began planning a recovery strategy based on the theory that the economy had shrunk at an alarming 3.8 percent annualized rate. After several rounds of revisions, the correct figure seems to have been an 8.9 percent annual rate. That’s on a par with confusing a mild recession with a gigantic one.

Even the smartest theoretical or policy work is basically valueless if it’s based on flawed or misleading data. And yet in the hierarchy of the profession, actually wrangling data is a relatively low-status undertaking compared with building elaborate edifices on even the shakiest of foundations. Harry Truman famously wished out loud for a “one-handed economist” who would offer concrete advice with less hedging and trimming, but, actually, overconfidence about bad data is the bigger problem. Absent laboratory conditions, it’s very difficult to obtain precise measurements of key figures, leaving us all too often with big theories based on poor numbers. And when theories move out of the ivory tower and into the policy realm, the problem gets worse as pressure for timeliness and relevance escalates.

To an extent, these are problems we’ll never overcome. But they underscore the importance of the unglamorous work of America’s government statistical agencies. Finding ways to obtain and use better data are ultimately key to developing better theories and better policies. The ATUS didn’t exist at all until 2003, which is why it hasn’t traditionally been used for productivity or other secondary statistics. But it appears to give not just slightly more precise estimates of how much people work but a totally different picture of large-scale economic trends and the nature of the business cycle. Neither economists nor policymakers talk nearly enough about ways to further improve the data available. Dull as government statistics sound to most people, ultimately they’re the basis for choices that make huge differences to the lives of millions.


War Stories

The Right Target

Why Obama’s airstrikes against ISIS may be more effective than people expect.

The One National Holiday Republicans Hope You Forget

It’s Legal for Obama to Bomb Syria Because He Says It Is

I Stand With Emma Watson on Women’s Rights

Even though I know I’m going to get flak for it.

Should You Recline Your Seat? Two Economists Weigh In.


It Is Very, Very Stupid to Compare Hope Solo to Ray Rice

Or, why it is very, very stupid to compare Hope Solo to Ray Rice.

Building a Better Workplace

In Defense of HR

Startups and small businesses shouldn’t skip over a human resources department.

Why Is This Mother in Prison for Helping Her Daughter Get an Abortion?

Politico Wonders Why Gabby Giffords Is So “Ruthless” on Gun Control

Sept. 23 2014 4:45 PM An Up-Close Look at the U.S.–Mexico Border
  News & Politics
Sept. 23 2014 6:40 PM Coalition of the Presentable Don’t believe the official version. Meet America’s real allies in the fight against ISIS.
Sept. 23 2014 2:08 PM Home Depot’s Former Lead Security Engineer Had a Legacy of Sabotage
Sept. 23 2014 1:57 PM Would a Second Sarkozy Presidency End Marriage Equality in France?
  Double X
The XX Factor
Sept. 22 2014 7:43 PM Emma Watson Threatened With Nude Photo Leak for Speaking Out About Women's Equality
  Slate Plus
Political Gabfest
Sept. 23 2014 3:04 PM Chicago Gabfest How to get your tickets before anyone else.
Brow Beat
Sept. 23 2014 8:38 PM “No One in This World” Is One of Kutiman’s Best, Most Impressive Songs
Future Tense
Sept. 23 2014 1:50 PM Oh, the Futility! Frogs Try to Catch Worms off of an iPhone Video.
  Health & Science
Bad Astronomy
Sept. 23 2014 11:00 AM Google Exec: Climate Change Deniers Are “Just Literally Lying”
Sports Nut
Sept. 23 2014 7:27 PM You’re Fired, Roger Goodell If the commissioner gets the ax, the NFL would still need a better justice system. What would that look like?