How will we know if Nate Silver was right?

How Will We Know If Nate Silver Was Right?

How Will We Know If Nate Silver Was Right?

A mathematician's guide to the news.
Nov. 7 2016 3:24 PM

Who Will Be America’s Next Top Modeler?

I have a plan to figure out which poll aggregator is the most accurate. Will Nate Silver and his cohort play along?

Nate Silver.
Nate Silver’s site FiveThirtyEight has given Donald Trump a 1-in-3 shot of winning the presidency.

Photo illustration by Slate. Photos by Slaven Vlasic/Getty Images, Thinkstock.

When we wake up Nov. 9, will we really know who won?

Not the presidential election—that we’ll know, barring the Evan McMullin Eschaton. I’m talking about what the campaign nerds are really focused on right now: the battle among the polling aggregators, who in the wake of Nate Silver’s successes in 2008 and 2012 have sprung up all over the internet like a scatter plot of toadstools after an autumn rain.

Advertisement

Every poll aggregator, from the famous (Sam Wang, the New York Times’ Upshot, Drew Linzer at Daily Kos) to the more obscure stat-hipster types (Andy Hoagland, Pierre-Antoine Kremp) agrees Donald Trump is unlikely to win.* But how unlikely? That’s where consensus melts into numerical discord. Wang says it’s basically over. As I write this, he gives Trump less than a 1 percent chance. The Upshot gang is more conservative, saying Trump still has a 16 percent chance of overcoming his deficit in the polls. And Nate Silver? Liberals have to lie down with a cold compress on their foreheads whenever they refresh his page, because he still sees Trump as having about a 1 in 3 shot of winning. In other words, Trump has as good a chance of gaining the White House as Jose Altuve does of getting a hit in a typical at-bat. And Jose Altuve gets a lot of hits.

If you think these disagreements must take place in the neutral emotional tone of an algebra textbook, you don’t know math people. We can be feisty. On Sunday, Ryan Grim of the Huffington Post accused Nate Silver of jiggering his estimates to make the race look closer, all the better to harvest the clicks of worried Democrats and hopeful Trumpists. He even called Silver a “pundit”—the worst insult imaginable in these quanty circles. The usually mild-mannered Silver returned fire:

He also called it “not defensible” to see Clinton as a 99 percent favorite, as Sam Wang does.

Who will turn out to be right, Nate Silver or his critics? That’s where things get sticky. Silver, after all, is telling us that Clinton might win and Trump might win. Can he even be wrong? Ezra Klein says no:

Advertisement

But Klein actually is a pundit. How does this look from a mathematician’s perspective?

I’d say right and wrong aren’t the words we should be using. I prefer better and worse. If Trump wins, for instance, Sam Wang isn’t exactly wrong—he admits there’s a nonzero chance that’ll happen! But he definitely comes out looking worse than Nate Silver does.

And if Clinton holds on, is everybody equally right? Not necessarily. Silver’s uncertainty is unusually strong in both directions. Wang thinks there’s only a negligible chance of Clinton getting more than 350 electoral votes, while Silver sees that as unlikely but plausible. If Clinton blows Trump out, that too makes Silver look better than Wang. For instance, would you bet, at even odds, that Clinton will wind up with between 300 and 339 electoral votes? Sam Wang would tell you to take that bet. But Silver would warn you away: His model gives that result less than 30 percent of the time. If Clinton ends up in that range, you can think of it as a rebuke to Silver’s uncertainty. Silver made a similar point Sunday night:

This isn’t just a thought experiment: You could put down 52 cents at PredictIt right now for a bet that pays you a dollar if the winner of the election, whether it’s Clinton or Trump, gets between 300 and 339 votes, indicating that the betting market considers this a roughly even bet.

Advertisement

Here’s one quick and dirty way we could rate the aggregators after the fact. Every one of them is offering what’s called a probability distribution: For each possible outcome of the election, they’re telling us how likely they think that outcome is. For instance, Sam Wang says there’s about a 2.7 percent chance that Clinton will pick up exactly 323 electoral votes. (I wasn’t able to find their predictions in numerical form, so all these numbers are eyeballed from the bar graphs on the aggregators’ websites. By the time you read this, those graphs may have changed.) Silver, who sees a much broader range of outcomes as plausible, doesn’t think any individual outcome is that probable; he gives 323 electoral votes just under a 1 percent chance of happening. The Upshot gives it about a 1.7 percent chance. Andy Hoagland thinks HRC323 is the most likely outcome of all, assigning it a 2.53 percent chance of coming to pass. (The handsome visuals were built by Mike Cisneros.) So in the 323 timeline, we might rank the aggregators Wang, Hoagland, Upshot, Silver. Even though Silver considers 323 electoral votes one of the most likely outcomes, he gets punished for not making an aggressive enough bet.

If Clinton sweeps the table and wins 400 electoral votes, it’s a different story. Wang gives that a zero percent chance of happening and comes in last. The Upshot and Nate Silver both consider that about a 1 in a 1,000 shot. Hoagland saw that happen just once in 20,000 simulations. So Silver and Upshot tie for first, with Hoagland far behind.

This isn’t the only way you could rank the aggregators—just the easiest. (Pros are often inclined to rate predictors using a Brier score, which requires a bit more computation) Of course the quick-and-dirty ranking system isn’t perfect; for instance, a predictor who went all in on Hillary getting 323 electoral votes, insisting there’s a 100 percent chance of that being the case, might win our contest if he or she ends up being right. But we would still say the predictor was more lucky and reckless than good.

The outcome of the presidential race depends on the electoral votes, but if I’m going to declare someone America’s Next Top Modeler, I want him or her to get the map right, too. So here’s my proposal. The poll aggregators should all release their probability distributions for the whole electoral map. Right now, only Andy Hoagland is doing this. His most likely map, which occurs in 0.8 percent of his simulations, has Clinton winning Florida, North Carolina, and Nevada and Trump taking the stray electoral votes in Maine’s 2nd Congressional District and Nebraska’s 2nd Congressional District. (It assumes the Washington electors bound to Clinton will actually vote for her; math just can’t account for some things.)

Nate Silver, Upshot crew, Drew Linzer: Will you release this data, too? Your bragging rights—with me, at any rate—depend on it.

Correction, Nov. 8, 2016: This article originally misidentified the person behind a polling model. It belongs to Andy Hoagland, not Mike Cisneros; Mike Cisneros visualizes Hoagland's work. (Return.)