Why So Much Economics Data Is Garbage

Commentary about business and finance.
Jan. 7 2013 3:05 PM

Economists Use Shoddy Data

All too often, economists’ and policymakers’ theories are based on terrible statistics.

How can we get better data?
How can we get better data?

iStockphoto/Thinkstock.

Economists are a smart bunch, and in San Diego over the weekend for the annual meeting of the American Economic Association lots of impressive analytical chops were on display. Sometimes, though, the analysis seems to badly overreach the data. A well-attended panel discussion about whether economics is hopelessly riven by ideological disagreement, for example, proceeded even though the IGM Chicago survey it’s based on isn’t even close to being a statistically valid survey. And the issue recurs even outside the realm of such “fun” discussions and infects much more sober research. No grand theory is going to be much better than the data it’s based on, and the underlying quality is often distressingly poor.

A worse-attended, dramatically lower-charisma discussion occurred the following day about a dully titled paper on “Cyclical Variation in Productivity Using the ATUS” by Michael Burda, Daniel Mamermesh, and Jay Stewart.

The point of the paper is that theories of the macroeconomy often depend crucially on ideas about the direction of productivity in the labor force. And to know how productive workers are on a per-hour basis, you need to know how many hours people work. This, in turn, is a messy business. The data traditionally come from two sources. One, the Current Employment Survey (CES), is a survey of employers and thus misses the self-employed and can’t tell how many people are working multiple jobs. For the vast majority of its history the CES also missed supervisory workers. The other, the Current Population Survey (CPS), asks individuals and is thus more comprehensive. But asking people what they did last week often generates inaccurate answers.

Advertisement

A better solution is to use the American Time Use Survey (ATUS), which asks people to keep a real-time diary, ensuring that numbers add up to 24 hours in a day and reducing memory error.

ATUS data has been around for a while, but as with the CES and CPS, its hours-worked series isn’t really used for anything despite being generally considered more accurate. What Burda, Hamermesh, and Stewart did was recalculate hourly productivity data with ATUS hours worked as the denominator. The scale of the change here is not enormous—the data series differ, but not all that much—but one interesting thing did happen. According to the standard data, productivity used to fall during recessions but has risen during recent recessions. According to the ATUS data, productivity still falls during recessions. The authors disclaim any ambitions to become macroeconomic theorists but did draw attention to a key point. The traditional view had been that during recessions employers engage in “labor hoarding,” hanging on to more workers than necessary in order to avoid wasteful firing and rehiring. More recently, hoarding seemed to have gone away. Firms instead took recessions as opportunities to trim the fat, firing low-quality workers and raising average productivity. This apparent switch in firm behavior has been thought by many to relate to the rise of “jobless recoveries.” Except now it looks like the change may never have happened.

So a small fussy point ends up undermining a big-picture argument. And it happens not just in academia, but in policy circles as well. The Obama administration, for example, took office in January of 2009 and began planning a recovery strategy based on the theory that the economy had shrunk at an alarming 3.8 percent annualized rate. After several rounds of revisions, the correct figure seems to have been an 8.9 percent annual rate. That’s on a par with confusing a mild recession with a gigantic one.

Even the smartest theoretical or policy work is basically valueless if it’s based on flawed or misleading data. And yet in the hierarchy of the profession, actually wrangling data is a relatively low-status undertaking compared with building elaborate edifices on even the shakiest of foundations. Harry Truman famously wished out loud for a “one-handed economist” who would offer concrete advice with less hedging and trimming, but, actually, overconfidence about bad data is the bigger problem. Absent laboratory conditions, it’s very difficult to obtain precise measurements of key figures, leaving us all too often with big theories based on poor numbers. And when theories move out of the ivory tower and into the policy realm, the problem gets worse as pressure for timeliness and relevance escalates.

To an extent, these are problems we’ll never overcome. But they underscore the importance of the unglamorous work of America’s government statistical agencies. Finding ways to obtain and use better data are ultimately key to developing better theories and better policies. The ATUS didn’t exist at all until 2003, which is why it hasn’t traditionally been used for productivity or other secondary statistics. But it appears to give not just slightly more precise estimates of how much people work but a totally different picture of large-scale economic trends and the nature of the business cycle. Neither economists nor policymakers talk nearly enough about ways to further improve the data available. Dull as government statistics sound to most people, ultimately they’re the basis for choices that make huge differences to the lives of millions.

TODAY IN SLATE

Politics

Talking White

Black people’s disdain for “proper English” and academic achievement is a myth.

Hong Kong’s Protesters Are Ridiculously Polite. That’s What Scares Beijing So Much.

The One Fact About Ebola That Should Calm You: It Spreads Slowly

Operation Backbone

How White Boy Rick, a legendary Detroit cocaine dealer, helped the FBI uncover brazen police corruption.

A Jaw-Dropping Political Ad Aimed at Young Women, Apparently

The XX Factor
Oct. 1 2014 4:05 PM Today in GOP Outreach to Women: You Broads Like Wedding Dresses, Right?
Music

How Even an Old Hipster Can Age Gracefully

On their new albums, Leonard Cohen, Robert Plant, and Loudon Wainwright III show three ways.

How Tattoo Parlors Became the Barber Shops of Hipster Neighborhoods

This Gargantuan Wind Farm in Wyoming Would Be the Hoover Dam of the 21st Century

Moneybox
Oct. 1 2014 8:34 AM This Gargantuan Wind Farm in Wyoming Would Be the Hoover Dam of the 21st Century To undertake a massively ambitious energy project, you don’t need the government anymore.
  News & Politics
Politics
Oct. 1 2014 7:26 PM Talking White Black people’s disdain for “proper English” and academic achievement is a myth.
  Business
Buy a Small Business
Oct. 1 2014 11:48 PM Inking the Deal Why tattoo parlors are a great small-business bet.
  Life
Outward
Oct. 1 2014 6:02 PM Facebook Relaxes Its “Real Name” Policy; Drag Queens Celebrate
  Double X
The XX Factor
Oct. 1 2014 5:11 PM Celebrity Feminist Identification Has Reached Peak Meaninglessness
  Slate Plus
Behind the Scenes
Oct. 1 2014 3:24 PM Revelry (and Business) at Mohonk Photos and highlights from Slate’s annual retreat.
  Arts
Brow Beat
Oct. 1 2014 9:39 PM Tom Cruise Dies Over and Over Again in This Edge of Tomorrow Supercut
  Technology
Future Tense
Oct. 1 2014 6:59 PM EU’s Next Digital Commissioner Thinks Keeping Nude Celeb Photos in the Cloud Is “Stupid”
  Health & Science
Science
Oct. 1 2014 4:03 PM Does the Earth Really Have a “Hum”? Yes, but probably not the one you’re thinking.
  Sports
Sports Nut
Oct. 1 2014 5:19 PM Bunt-a-Palooza! How bad was the Kansas City Royals’ bunt-all-the-time strategy in the American League wild-card game?