FiveThirtyEight vs. the Upshot: Who should you trust to forecast the 2016 presidential election?

Who Should You Trust to Forecast the Presidential Election, FiveThirtyEight or the Upshot?

Who Should You Trust to Forecast the Presidential Election, FiveThirtyEight or the Upshot?

Who's winning, who's losing, and why.
Aug. 2 2016 6:34 PM

Which Election Forecast Should You Trust?

FiveThirtyEight and the Upshot both aggregate a bunch of polls. Here’s why their predictions are so different.

Nate Silver attends Tribeca Talks / ESPN Sports Film Festival: Data Lab for Storytelling during the 2015 Tribeca Film Festival at Spring Studio on April 20, 2015 in New York City.
Nate Silver’s FiveThirtyEight and the New York Times’ the Upshot offer two different sets of numbers. Above, Silver at the Tribeca Film Festival in New York City in 2015.

Dave Kotinsky/Getty Images

What are the chances that Donald Trump will be the next president of the United States? If you’ve been following Nate Silver’s FiveThirtyEight, you’ve seen those chances steadily increase over the last few weeks, with Trump rising from a 23-percent longshot in the site’s “polls-only forecast” to a 50–50 bet to win the presidency. As I write this on Aug. 2, in the aftermath of a sizable post-convention bounce for Hillary Clinton, the Republican nominee’s chances have dropped back down to 34 percent. The New York Times’ the Upshot has been a bit more consistent in its prognostications; its current estimate that Trump has a 26-percent shot to reach the White House is about on par with its recent projections.

Though both sites aggregate polls to arrive at their predictions, their outcomes differ because they use different methods to assess those polls. Which election forecaster is right for you? If you want to keep a close eye on the latest trends, go with Nate Silver and FiveThirtyEight. If you’d prefer to avoid any sudden jolts, go with the Times.

Advertisement

Silver does an excellent job spelling out the methodology for his polls-only forecast. (I’m focusing on the polls-only forecast as opposed to FiveThirtyEight’s “now-cast” and “polls-plus” forecasts because it offers the closest parallel to the Upshot’s methodology.) First, the FiveThirtyEight team collects a broad array of national and state polls. The polls are then weighted based on FiveThirtyEight pollster rankings, their sample sizes, and their recency before they are averaged in each state.

In Nevada, for instance, the polling average currently shows Clinton beating Trump by just a tenth of a percentage point, 42.7 percent to 42.6 percent. Those numbers are adjusted based on both historical and current data to account for the effects of a bunch of different factors, including the presence of third-party candidates, biased polling, and national and state polling trends. The latter adjustment is particularly important when there are long gaps between polls and is helpful for reconciling poll results that seem like outliers. “That’s the part that says, well, we know that if Wisconsin is plus-6 for Clinton then Minnesota probably can’t be pro-Trump even though there was a poll that came out there that had Trump ahead,” Silver explains.

After all those adjustments, the adjusted average shows Trump beating Clinton just slightly in Nevada: 44.2 percent to 43.3 percent. “If there were 10 polls every day in every state, then you wouldn’t have to do anything fancy,” Silver says of the polling trend adjustments. “But the point is that it’s kind of naïve to assume that if the national polls, say, have broken and moved 5 points toward Trump, that that hasn’t happened in Ohio, in Pennsylvania, and Florida whether or not we have polls there.”

The next part of Silver’s method essentially constructs the context for the election, splitting undecided voters between the candidates and projecting the national popular vote using both national and state polls. FiveThirtyEight then uses statistical regressions that calibrate current poll results with those from the previous two elections, geography, and demographic variables such as race and religion. Those regressions are then used to simulate the election more than 20,000 times each day, and those simulations in turn produce both the probabilities of each candidates winning individual states and winning the election overall.

Advertisement

The result: As of right now, FiveThirtyEight says Clinton has a 59.5-percent chance of winning Nevada to Trump’s 40.5 percent. That’s the case even though the two candidates were roughly even in the polling average and Trump actually beat Clinton in the adjusted polling average. Just two days ago, FiveThirtyEight gave Trump a 54.4-percent chance of winning Nevada. The odds have now shifted dramatically despite the fact that there’s been no recent polling in Nevada. That’s because Silver and co. use national polling trends to adjust their individual state models. Thanks to those national trends, Clinton’s chances of winning the general election have risen to 66 percent compared to just 51 percent two days ago.

The Upshot explains its methodology in less detail, though Josh Katz’s write-up of the presidential prediction model does go over the general principles. Unlike FiveThirtyEight, the Upshot does not weight its polls based on pollster reliability, focusing exclusively on sample sizes and recency. The Upshot’s regressions are also simpler than FiveThirtyEight’s, relying more on past election data than the demographic information and regional groupings that Silver incorporates. The model is also designed to be “slow to move.” Here’s Katz’s explanation:

Our model starts with a weighted average of polls in each state, giving polls conducted more recently and polls with a larger sample size a greater weight in the average. At this point in the election cycle, it uses a longer time window to calculate these averages. This steadiness means it is more stable and less inclined to chase after the most recent poll. But it also means it is slower to react to developing trends, such as recent polling that may indicate a tightening race. This can be a feature or a bug, depending on your perspective. Preconvention polls are informative, but history suggests that it is a mistake to place too much emphasis on a week’s worth of polling.

What does that mean in practice? In Nevada, the Upshot rates Clinton’s chances of winning at 72 percent, just more than 12 percentage points higher than FiveThirtyEight’s polls-only estimate. Since the Upshot doesn’t provide detailed state-level data—as opposed to FiveThirtyEight, it doesn’t have a dedicated page for Nevada—we can only speculate about why the Times’ model is sunnier about Clinton’s chances. It’s reasonable to speculate, though, that the Upshot likes Clinton in Nevada because that would be in line with recent election results.

Advertisement

Indeed, even before Clinton’s post-convention bump, the Upshot projected her to win most of the key swing states Obama won in 2012—Nevada, Iowa, Ohio, and Florida—and rated New Hampshire as a tossup. This is a result one might intuitively expect from a model more rooted in past results. FiveThirtyEight’s forecast, on the other hand, had Clinton losing every single one of those states to Trump—including Nevada. But since Clinton’s bounce, FiveThirtyEight’s model has swung around to predicting Clinton will sweep those states, Nevada included.

The differences between the two forecasting methods are perhaps best illuminated by a set of histograms that show the probability of Hillary Clinton winning a certain number of electoral votes.

The Upshot’s histogram is taller and well-centered, which suggests it’s more confident than FiveThirtyEight is that Clinton will win more than 300 electoral votes in November (270 are needed to win). The FiveThirtyEight histogram also suggests that Clinton winning more than 300 electoral votes is the likeliest outcome, but its histogram is shorter and wider, indicating a wider range of possible outcomes and higher probabilities for a Trump victory.

At least for the moment, then, if you want to believe that Donald Trump has a decent shot to win the presidency, head over to FiveThirtyEight. If that’s a world you don’t want to live in, read the New York Times.