The Phony Science of Predicting ElectionsWho'll win in November? The experts' guess is as good as yours.

Reader Response from The Fray:

A core value of election forecasting research is accuracy. I would commend this as well to Karl Eisenhower and Pete Nelson, and to the editors of Slate. For anyone who wants to know the details about how forecasting is done--and this would include Eisenhower and Nelson, who quite evidently did not do their homework--I would recommend that they read Before the Vote (Campbell and Garand) from Sage Publications, The American Campaign (Campbell) from Texas A&M University Press, and Do Campaigns Matter? (Holbrook) from Sage. The Eisenhower and Nelson piece on election forecasting is replete with errors and cheap shots. Let's get specific. Here are the Eisenhower and Nelson errors, point by point:

1. They say that the forecasts are simply based on data-fitting to past elections. Nonsense. Each of the seven independent models that published in Before the Vote conduct out-of-sample tests of their model. For the non-statistically inclined, the model is estimated without an election and then the values for that excluded election are inserted to generate what the "forecast" would have been. This is done for each election year. This is not predicting "the outcomes they used to create the models in the first place." Moreover, the strength of models are evaluated in a variety of other ways. There are statistical techniques (robust regression) that reveal whether models are particularly dependent on a few elections. I examined my model throughout various points in the campaign and with different economic indicators, essentially multiple tests of the model. Several forecasters in Before the Vote discuss, at length, how to evaluate confidence in the models and certainty in any particular forecast.

2. Eisenhower and Nelson say that the forecasters predict the obvious. At least E & N's hindsight is 20-20. First, in retrospect, all election results look predictable. However, when you compare the forecasts to polls or pundits, the record of the forecast models is terrific. As is documented in Before the Vote, the seven forecasts of the 1996 election made 2 to 3 months before the election were at least as accurate as a group as the national polls reported in USA Today on election day. Are the forecasting models perfect? No. And as a result they (like the polls or pundits) may miss very close elections, such as the 1960 and 1968 elections. However, when it comes to reliably accurate forecasts of the national vote split, the hard record clearly indicates the greater accuracy of the models.

3. Eisenhower and Nelson snidely suggest that the models have "ducked" the hard calls, most notably Truman's 1948 comeback. This, again, is nonsense. All of the models include all elections in which data for their predictor variables is available. If a forecaster excluded a year just to help his model, someone else would estimate the model with the year included. There is some healthy competition here and the data that we are all using is publicly available to anyone who wants it. Moreover, four of the seven models included in Before the Vote include the 1948 election in their analyses (Campbell, Abramowitz, Norpoth, and Holbrook). Those who do not include 1948 cannot do so because they need data that was not collected at that time. Again, Eisenhower and Nelson should read before they write.

4. E&N criticize the forecasting models because they use polls ingenerating their forecasts. Well some do and some do not. I would think that the use of polls in forecasting would make the forecasts more credible to E&N, but consistency, like accuracy, does not seem high on their list of values. My perspective of forecasting, using polls in the forecast, is that it is a more sophisticated way of reading polls. If a candidate is ahead by 60-40 at a particular point in the campaign, should we take this at face value or try to put it in some historical context and the context of the current economic climate? I think that an intelligent observer would do the later and that is what several of the forecasting models do. For the models that do not use polls, I think that they are taking on the tougher task of trying to forecast with one hand behind their back. However, if they can obtain good measures of what tends to influence the electorate, then more power to them.

5. E&N criticize the forecasters for making multiple forecasts throughout the election year. The problem here is that E&N personalize forecasting. Strictly speaking, the forecast models make a forecast, not the guy who constructed the model. So, I have a model estimated predictors available in July, with predictors available after the conventions, with predictors available on Labor Day. My analysis indicates that the Labor Day prediction is the most accurate, and so that is the one that I rely on. Others have models that also have predictions at different time points. So long as the specific model is cited as the one producing the forecast, I see no problem other than the fact that it may create some confusion among those who are not paying close attention Apparently, Eisenhower and Nelson can be counted in this group.

6. E&N criticize forecasters for picking the polling data that they use in their forecasts. Pay attention boys. Forecasters attempt to be as consistent as possible. If their models are estimated over time using Gallup data, they will use Gallup data in generating a forecast rather than move to CBS or someone else's data.

7. E&N say that forecasters "shrug off errors" and are proud of dead-on predictions. Again, the forecast is from a model not the forecaster. Lewis-Beck had one model that was not accurate in 1992 and a different model that was accurate in 1996. Ray Fair's model in 1992, a model not using poll data, had a large error and he revised his model for 1996.The point is that the models are not perfect and we all learn from errors. That's the way we get stronger models and errors are duly noted, not "shrugged off." By the way, though he has a chapter in the book, Before the Vote is not Lewis-Beck's book. E&N, at least read the cover.

8. E&N accuse the forecasters of "tweaking the numbers" and altering the forecasting models. As noted above, some models are revised in light of errors or rethinking the model to improve it. However, as anyone who reads Before the Vote, there were no significant changes in any of the models used in 1996 and 1996 other than adding another year in updating the models.

9. E&N say that forecasters add loopholes, or variables specific to an election, to unfairly fit the models to election outcomes. They specifically accuse Holbrook of fitting 1964 and 1972 because of extremist candidates Goldwater and McGovern. I have no idea where they got this idea from. I searched Holbrook's article and could find nothing that could even be misinterpreted in this way.

10. E&N in their point 10 claim that elections across a long expanse of time cannot be compared. Well this is an interesting idea, but in the end it is one that is open to testing. They draw a questionable analogy of comparing elections over time in forecasting to determining whether Babe Ruth or Mark McGwire was a better hitter. I would say that a better analogy is in comparing teams in mid to late season and whether they could survive to the World Series. If a team is up by twelve games with twenty left to play, and has evidence of solid pitching and hitting (no key injuries) it will probably win the pennant. In any case, forecasting rests on facts not analogies.

As to Eisenhower and Nelson's final three points: (1) The forecasting models have been quite accurate without subjective indicators of campaign quality. (2) They accuse the models of being based on truisms, which is to say obvious indicators. I plead guilty. We are using common-sensical indicators and examining them over history to obtain more precise expectations of an election's results. What is the problem? (3) The fact that elections can be forecast accurately does not mean that campaigns do not matter. To the contrary, I assume that they do matter and that the fundamentals, like the economy, are the ammunition of good campaigns.

In the end, for anyone who is interested, forecasting rests on pretty sensible foundations. The Eisenhower and Nelson piece, however, is not worth the cyberspace it was digitized on.

P.S. If Eisenhower and Nelson had read the Before the Vote book, they would not only not have attributed it to Michael Lewis-Beck but would not have incorrectly listed my affiliation as LSU. It is noted at least four times in the book that I am affiliated with the University at Buffalo, SUNY.

--Prof. Jim Campbell,
Professor of Political Science,
University of Buffalo, SUNY

(To reply, click here.)



We don't regret pointing out why Lewis-Beck and Campbell should be modest about the accomplishments of their models, but we do regret a superficial editing error that confused the names of their books. We also don't regret our criticism of forecasters' use of ad hoc variables, but we do regret citing Holbrook's model as an example. Holbrook used an extremism variable to analyze determinants of elections, not to forecast them. A better example would have been Campbell's May 1992 article, "Forecasting the Presidential Vote in the States" (in the American Journal of Political Science), in which he used ad hoc variables to fit his model to the data and make a prediction from it. Finally, we regret identifying Campbell with his previous employer, Louisiana State University (as his publisher's Web site does), rather than with his current employer, the University at Buffalo, SUNY.

None of these points affects the substance of the piece. The models have a very small sample size. Most of them use poll numbers of some kind (which have a built-in error) as variables. They make a dubious assumption that the statistical relationship between the economy and elections is not going to change in half a century. Campbell's belief that using out-of-sample "pseudo-forecasts" dispenses with the retrospective estimation issue misses the point. The Super Bowl model performs well out-of-sample too.

Campbell claims that "when it comes to reliably accurate forecasts of the national vote split, the hard record clearly indicates the greater accuracy of the models." Let's look at that record. Campbell's model was developed in 1990, meaning that he has attempted to predict just two elections. Although he got within a percentage point in 1992, his 1996 prediction, made with the benefit of September polling data, was off by almost four points. What difference does four points make? A lot according to Campbell. In October 1996, he told the Buffalo News that, assuming this forecast was correct, the Democrats would likely recapture the House.

What the forecasting research can reasonably be said to demonstrate is that early horse-race poll numbers don't mean much. Voters start paying attention to the election around mid-summer, and when they do, the economy and overall satisfaction with the incumbent administration become important factors. We leave it to readers to judge whether the profundity of this observation justifies the publicity it has received. In any case, it doesn't justify feeding reporters half-baked, down-to-the-decimal-point predictions in May.

--Karl Eisenhower and Pete Nelson

(To reply, click here.)

(6/14)

You are taking political science far more seriously than political science takes itself. Forecasting is widely acknowledged to be a parlor game in the discipline, sort of a Kevin Bacon for the statistical class. For example, if you look at the seat-swing models for congressional control, no-one got the 1994 election correct --not even close. The best predictor of Bill Clinton's job approval and political performance is the number of songs in the country music top 10 that deal favorably with adultery (really). Relax!

--Old Prof

(To reply, click here.)


The nerve of these scientists to try and predict the next elections. Don't they know that prognostication is limited to the high priests of journalism?

Quick! Knock them down before they gain more credibility! People might start to ignore us!

--Merrill Guice

(To reply, click here.)


This article is chock full of bogus criticisms: The scientists' models get closer to the truth as an election nears (shouldn't they?); they're the least reliable when the election is most closely contested (what else would you expect?); they use past data to predict the future (I can't imagine what else they'd do); scientists incorporate new data to "tweak" the models between elections (should they ignore the new--and most pertinent--data instead?).

A particularly comic criticism is that scientists are foolish to want more data. "This assumes that subsequent elections would clarify rather than complicate the range of data..."? No, it doesn't. Scientists want more data because, by definition, more data is more information about the true state of the world. If more data yield a more complicated picture, so be it. We're still closer to the truth. Scientists are not partial to the haphazard method of inference this article implies. Eisenhower and Nelson should rethink their argument and the vehemence with which it's made.

--Tom

(To reply, click here.)

(6/1)

• Robinson: A Little Worried About the Meltdown
• Khaled Hosseini: Sen. McCain, Am I a Pariah?
• Ombudsman: A Puff Piece About the Obamas?
• King: The Anatomy of an Assault