The Wisdom of Crowds?A single economic forecast is usually wrong. But groups of economic forecasts are often just as mistaken. Why?
Posted Saturday, Aug. 9, 2008, at 6:44 AM ETWhen people discover that I am an economist, they rarely ask me for my views on subjects that economists know a bit about—such as how to respond to climate change or pay less at a supermarket. Instead, they ask me what will happen to the economy.
PRINT
DISCUSS
E-MAIL
RSS
RECOMMEND...
Facebook
MySpace
Mixx
Digg
Reddit
del.icio.us
Furl
Ma.gnolia
Sphere
StumbleUponWhy is it that people won't take "I don't really know" for an answer? People often chuckle about the forecasting skills of economists, but after the snickers die down, they keep demanding more forecasts. Is there any reason to believe that economists can deliver?
One answer can be gleaned from previous forecasts. Back in 1995, economist and Financial Times columnist John Kay examined the record of 34 British forecasters from 1987 to 1994, and he concluded that they were birds of a feather. They tended to make similar forecasts, and then the economy disobligingly did something else, with economic growth usually falling outside the range of all 34 forecasters.
Perhaps forecasting technology has moved on since then, or the British economy is unusually unpredictable? To find out, I repeated John's exercise with forecasts for economic growth for the United Kingdom, United States, and Eurozone over the years 2002-08, diligently collected at the end of each previous year by Consensus Economics.
The results are an eerie echo of John Kay's: For 2004, for example, 20 out of 21 nongovernmental forecasts made in December 2003 were too pessimistic about economic growth in the United Kingdom. The Pollyannas of the U.K. treasury were more optimistic than almost any commercial forecaster and closer getting their forecast right. So, one might suspect that systematic pessimism is to blame.
But, no, in 2005, the economy grew more slowly than 19 out of 21 forecasters had expected at the end of the previous year. The Pollyannas of the U.K. treasury were yet again more optimistic than anyone and thus more wrong than anyone. A year later, all but one of the forecasters were too pessimistic again. Yet at the end of 2001, three-quarters of the forecasters were too optimistic about 2002.
2003 is an interesting anomaly: the one year for which the average U.K. forecast turned out to be close to reality but also the year where the spread between highest and lowest forecast was widest. The rare occasion that the forecasters couldn't agree happened to be the occasion on which they were (on average) right.
Recent U.S. forecasters have done a little better: The spread of forecasts is tighter, and the outcome sometimes falls within that spread. Still, five out of six were too pessimistic about 2003, almost everyone was too pessimistic about 2002, three-quarters were too optimistic about 2005, and nearly nine-tenths too optimistic about 2006. Perversely, the best quantitative end-of-year forecasts were made in December 2006, despite the fact that the credit crunch materialized eight months later to the surprise of almost everybody.
In the Eurozone, forecasting over the past few years has been so wayward that it is kindest to say no more.
The new data seem to confirm Kay's original finding that economic forecasters all tend to be wrong in the same way. Their incentives to flock together are obvious enough.
What is less clear is why the flight of the flock is so often thought to augur much—but then, some astrologers are also profitably employed.
The curious thing is that forecasters often have something useful to say, but it is rarely conveyed in the numerical forecast itself on which so much attention is lavished. For instance, in December 2006, forecasters were warning of the risks of an oil price spike, a sharp rise in the cost of credit, and a dollar crash. The quantitative forecasts are usually wrong and not terribly helpful when right, but forecasters do say things worth hearing, if only you can work out when to listen.
Happy Birthday, Smokey Bear
Are Gas Grills More Eco-Friendly Than Charcoal Ones?
He-Man: Briefs of Rage and Other Toy-Inspired Movies We're Dying To See
Kaus: Seven Possible Theories Explaining Palin's Resignation
The U.S. Embassy in Djibouti Cordially Invites You to a Fourth of July Cookout
The Week's Best Editorial Cartoons
Remarks from the Fray:
The answer to why groups of economists are no more often right than single economists is pretty easy: they're all using the same tools to generate their answers.
The "Wisdom of Crowds" functions not on the basis of experts following rational processes, but rather on the basis of people making irrational decisions and then aggregating the tiny bits of knowledge contained in those decisions to (hopefully) a more accurate picture of the whole.
--Xando
(To reply, click here.)
Conventional wisdom is, well, conventional. Professional economists, not wanting to be subject to ridicule by their peers, will look at existing trends and come to the conventional conclusion that these trends will continue. What else could they do?
Economies have as many variables and unknowns as the weather. I'd like to know how many weather forecasters predicted the existence and track of hurricane Katrina a month before it struck Mississippi and Louisiana.
What we're left with is SWAG -- scientific, wild-ass guesses.
--revrick
(To reply, click here.)
Tim's criticism appears to have essentially two points: That economic forecasts are either correct or incorrect; and that forecasters consistently make similar mistakes. Well, his data isn't too useful because it doesn't account for relativity.
Economics makes relative comparisons. Measuring an economic forecast in binary terms (hit/miss) is like scoring golf by "holes-in-one" (yes/no). This measurement isn't fair or helpful because it equally values huge inaccuracies and near-misses.
What if all those forecasts were within a fraction of a percent of the true economic measurement? What if they were no more accurate than random? For example, predicting 5% growth before a 4.5% measurement is MORE ACCURATE than predicting a 5% contraction (negative growth) before the 4.5% growth. Moreover, consistently coming within 10% of the mark is clearly useful because it helps reduce risk, especially relative to random. Eliminating risk may be impossible, but reducing risk is not.
--JayMM
(To reply, click here.)
Forecasting is hard, and not just for economists. […] Consider engineers. Not usually considered forecasters, but when they build a building they are, in effect, forecasting that it will not fall down. And they are pretty good--not perfect, as buildings (bridges, dams, etc) fail from time to time because an engineer made what could be called a forecasting error.
How do engineers do it? Bigger brains, perhaps? No. They typically build in huge safety buffers--100 percent or more is typical for a bridge. Well, if economists could forecast next year's GDP with that sort of error, they would never be wrong (and would never be useful, either).
Is there anybody who can forecast the future?
The only case I can think of is astronomers, who for thousands of years have been able to accurately forecast the position of the planets, eclipses, sunrises, and the like. Amazingly, they were able to do this even while using what we now know to be incorrect theories (notably, the earth-centered, but nonetheless successful from a forecasting perspective, Ptolemaic model). This is because, it turns out, the movements of the planets are pretty simple.
And ever since, forecasting has been ruined by this one success.
--lloyd667
(To reply, click here.)
If an economist came up with a completely different forecast than the great majority of economists, and he (or she) turns out to be right, he'll get a pat on the back. If he turns out to be wrong, he'll get fired (or laughed out of the next professional conference, which amounts to the same thing).
Economic forecasting is complex for a system as large as the United States, with an almost infinite number of factors that can potentially throw it off. No theoretical model can incorporate all of the variables in play, and so economists tend to lump themselves into one united camp on their forecasts (if no one else saw event "X" coming, it is not reasonable to ask me to predict it), While economics may be useful for many things, predicting the future is not one of them as long as we have our current set of tools.
--Bill N
(To reply, click here.)
(8/15)