If prediction markets are so great, why have they been so wrong lately?

The conventional wisdom debunked.
April 16 2008 7:18 AM

The Fizz-dom of Crowds

If prediction markets are so great, why have they been so wrong lately?

A cracked crystal ball
How accurate are prediction markets?

When political junkies first encountered prediction markets in the early 1990s, it was as if the opinion poll had been perfected. Academic studies showed that the markets—in which individuals bet real or fake money on futures contracts that expire when an election is over—consistently did a better job of predicting electoral outcomes than even exit polls. They've been an important part of Slate's political coverage for years.

But lately, prediction markets have been getting some big questions very wrong. Hollywood Stock Exchange and its cinema brethren are great at some things but have consistent trouble making accurate best supporting actor and actress predictions, including this year. In January, Slate's Dan Gross wrote about how the prediction markets failed during the crucial New Hampshire Democratic primary. Instead of beating the conventional wisdom, the prediction markets trailed it. The Intrade contract for Obama to win the New Hampshire primary rose as high as 95 cents; on the day that voters went to the polls in New Hampshire, Intrade still had Obama's chances of being the Democratic nominee as more than 70 percent. But as results favoring Hillary Clinton began to come in, the contracts became a dead heat. That's hardly a prediction—it's simply tracking the day's results.

On the evening of the California primary, the prices on InTrade indicated that Obama would win; he didn't. When Barry Ritholtz, who runs a quantitative research firm called FusionIQ and writes the popular Big Picture financial blog, wrote about New Hampshire in January, he reminded us that TradeSports had also failed to correctly predict the Democrats' recapturing the Senate in 2006, and that the Iowa Electronic Markets had Howard Dean as the favorite to win the Iowa caucus in 2004. Citing Ritholtz, the New York Times declared in February that "a little backlash has begun" against prediction markets.

So, what's going on? Is there something fundamentally wrong with this one-time darling of pop economists? Here's an examination of the leading pet theories for why prediction markets fail:

They're too small. That's one of Ritholtz's biggest gripes. Political-prediction markets, he wrote in January, "are thin, trading volumes are anemic, the dollar amounts at risk are pitifully small. Thus, these markets are subject to failure at times."

There's no question that tiny markets are bound to yield inadequate results. Yet by the usual standards of prediction markets, the trading on political futures markets has been pretty robust. In the days before the Iowa caucus, for example, several thousand dollars changed hands in the Iowa Electronic Markets, and tens of thousands of contracts were sold.

Yes, that level of participation is far lower than, say, the 2-billion-plus shares that are traded on the New York Stock Exchange most weekdays. But it's not so small as to alone make a prediction market inaccurate. Thomas Malone, who studies prediction markets at MIT's Center for Collective Intelligence, says that his rule of thumb is that only "somewhere around a dozen" participants are necessary to give a prediction market sufficient liquidity.

The stakes are too low. People betting on prediction markets are typically playing with small amounts of money (compared with big traders in capital markets) or with fake money. Therefore, one theory holds, they lack sufficient incentive to ensure that their calls are rigorous—they are lazy bettors.

This will ring true with anyone who's ever played online poker for fake money. The lack of any stings attached to losing leads players to stay in hands holding terrible cards; it brings down the quality of play and rewards reckless bets. A similar dynamic is at work in prediction markets, which is why conventional wisdom has long held that real-money prediction markets are more reliable than fake-money ones.

Yet that conventional wisdom has been challenged this year. InTrade is a real-money site, and it failed to accurately predict the California primary. Neither the play-money Hollywood Stock Exchange nor InTrade correctly predicted that Tilda Swinton would win the Academy Award for best supporting actress, but, somewhat surprisingly, the play-money site at least had Swinton in second place. And the incentives argument certainly can't explain recent failures in prediction markets, because nothing has changed: They offer the same minor incentives as they always have.

They're too slow to react to events. David Leonhardt made this argument in the New York Times. Citing Barack Obama's ups and downs in the Intrade market after the contests in Iowa (he won and went up), New Hampshire (he lost and went down), and South Carolina (he won and went up again), Leonhardt asserts that "the impact of each contest took surprisingly long to sink in."

Which only raises the question: compared with what? Leonhardt's yardstick is the stock price of drug maker Schering-Plough, which appears to react to market news almost instantaneously. That's true, but it's an individual equity, not a futures contract scheduled to expire at a prearranged date. As such, few would argue that its current price is intended to predict anything in particular about the company eight months down the road. One might just as usefully compare the political-prediction market with the prices of yachts or Beanie Babies. Moreover, there are counter-examples in which political futures seem to react pretty rapidly: On the day of the New Hampshire primary, a contract that Hillary Clinton would be the Democratic nominee traded as low as 18.1 cents and as high as 58.7 cents on the Iowa presidential market.

None of this is to suggest that all of the above theories are wrong, but they are each incomplete and unproven. I suspect that a large part of the problem has to do with expectations: Because prediction markets have been more accurate than other prognostications in the past, we crave them to be perfect. They aren't, and so long as they are built on bets in which no one has perfect information—like Oscar winners and political candidates—they never will be. Every bettor's perception of who will win is filtered through a myriad of inputs: a sense of who is the leader and who is the underdog; the influence of other people's bets; even something as mundanely human as whom we want to win. To get the maximum use out of them, we must—as with political polls—learn to read them in a discriminating, critical fashion. This year, that process seems to have begun.



Slate Plus Early Read: The Self-Made Man

The story of America’s most pliable, pernicious, irrepressible myth.

Rehtaeh Parsons Was the Most Famous Victim in Canada. Now, Journalists Can’t Even Say Her Name.

Mitt Romney May Be Weighing a 2016 Run. That Would Be a Big Mistake.

Amazing Photos From Hong Kong’s Umbrella Revolution

Transparent Is the Fall’s Only Great New Show

The XX Factor

Rehtaeh Parsons Was the Most Famous Victim in Canada

Now, journalists can't even say her name.


Lena Dunham, the Book

More shtick than honesty in Not That Kind of Girl.

What a Juicy New Book About Diane Sawyer and Katie Couric Fails to Tell Us About the TV News Business

Does Your Child Have Sluggish Cognitive Tempo? Or Is That Just a Disorder Made Up to Scare You?

  News & Politics
Sept. 29 2014 11:45 PM The Self-Made Man The story of America’s most pliable, pernicious, irrepressible myth.
Sept. 29 2014 7:01 PM We May Never Know If Larry Ellison Flew a Fighter Jet Under the Golden Gate Bridge
Dear Prudence
Sept. 29 2014 3:10 PM The Lonely Teetotaler Prudie counsels a letter writer who doesn’t drink alcohol—and is constantly harassed by others for it.
  Double X
Sept. 29 2014 11:43 PM Lena Dunham, the Book More shtick than honesty in Not That Kind of Girl.
  Slate Plus
Slate Fare
Sept. 29 2014 8:45 AM Slate Isn’t Too Liberal, but … What readers said about the magazine’s bias and balance.
Brow Beat
Sept. 29 2014 9:06 PM Paul Thomas Anderson’s Inherent Vice Looks Like a Comic Masterpiece
Future Tense
Sept. 29 2014 11:56 PM Innovation Starvation, the Next Generation Humankind has lots of great ideas for the future. We need people to carry them out.
  Health & Science
Medical Examiner
Sept. 29 2014 11:32 PM The Daydream Disorder Is sluggish cognitive tempo a disease or disease mongering?
Sports Nut
Sept. 28 2014 8:30 PM NFL Players Die Young. Or Maybe They Live Long Lives. Why it’s so hard to pin down the effects of football on players’ lives.