Followers of American politics are converging on a consensus that Republicans are likely to take control of the U.S. Senate following Tuesday’s election. But the real political quant nerds, as they refresh their browsers again and again while the returns roll in, will be focusing on a less settled question: How well will Nate Silver’s model perform? Will he, as he did in the 2012 presidential election, run the table and get every race right?
Where is the meta–Nate Silver who could make a principled mathematical prediction for how accurate Nate Silver is going to be? Who Nate Silvers the Nate Silvers?
Answer: Nate Silver himself.
Let me explain. As I write this on Monday, the Senate race that Silver’s algorithm is most uncertain about is the contest in Kansas, where the editor-in-chief of FiveThirtyEight gives independent Greg Orman a 52 percent chance of winning.* That is, if we ran through this Senate race 100 times—God help us—Silver estimates Orman would win 52 of them. So in those 100 elections, Silver’s prediction for Kansas would be correct 52 times and wrong 48 times.
North Carolina isn’t quite as close; Silver gives incumbent Democrat Kay Hagan a 71 percent chance of winning. In our imaginary 100-fold Senate race, Silver would get this race right 71 times, but would rack up 29 mistakes. So from Kansas and North Carolina alone, Silver is estimating that he’d make 48 plus 29 equals 77 blown calls.
And so on, through all the states in the FiveThirtyEight Senate forecast. The number of wrong predictions Silver would expect to make in 100 elections breaks down like this:
Kansas: 48
Iowa: 33
North Carolina: 29
Colorado: 28
Alaska: 28
Georgia: 26
Louisiana: 21
New Hampshire: 20
Arkansas: 6
Minnesota: 3
Kentucky: 3
West Virginia: 1
Total: 246
Technically, I ought to include rows for the 23 other states with contested Senate elections, too, but let’s be generous and grant that he’s going to get all of those right 100 times out of 100. (If Nels Mitchell somehow manages to become the first new member of the Idaho Democratic senatorial club since 1956, things have gone so far off-model that this calculation is probably not really relevant.)
So, as of early Monday afternoon, Silver predicts he’d get 246 predictions wrong if the election were run 100 times, an average of 2.46 per election. This number is what mathematicians call the expected value of the number of wrong predictions. It represents the number of Senate races, on average, Silver expects himself to be wrong about.
At first this might seem a bit contradictory—how can Nate Silver be predicting that Nate Silver is wrong? If you thought you were wrong about something, wouldn’t you just … think the opposite of what you think?
Not really. As a guy who wrote a book called How Not to Be Wrong, I had to become sort of an expert on this. The subtlety is well-captured by an old maxim of the philosopher Willard Van Orman Quine: “[A] reasonable person believes each of his beliefs to be true; yet experience has taught him to expect that some of his beliefs, he knows not which, will turn out to be false. A reasonable person believes, in short, that each of his beliefs is true and that some of them are false.”
Or, more succinctly: We always think we’re right, but we don’t think we’re always right. Nate Silver, who has always referred to his perfect accuracy in 2012 as a lucky accident, understands this point very well.
But not everybody does. If you’re a die-hard Silver hater, then, the seeming paradox provides an agreeable rhetorical opportunity. If he gets two or three states wrong, as he predicts he will, you can deliver an ex post facto dissection of the obvious political trends in those states that his cold equations failed to capture. And if he gets every state right, you can ding him for underestimating the number of calls he’d miss. Win-win!
That last ding actually isn’t that ridiculous. If Silver gets everything right, election after election, it may mean his predictions are improperly calibrated; in that case, when his method says 54 percent, maybe he ought to bump it up to 75 percent, and when it says 75 percent, maybe he ought to call it 90 percent.
But for now, I think Silver’s calibration looks pretty good. Somewhat lost in the glow of his bull’s-eye in the 2012 presidential election is the fact that he did miscall Senate races that year in Montana and North Dakota. My bet for 2014? Silver is likely to be about as wrong as he expects to be.
Of course, I could be wrong.
Update, Nov. 4, 11:20 a.m.: Commenter StevenJ writes: “Nate Silver’s model assumes positive correlation between the polling errors in the different Senate races. In other words, if he under- or over-estimates the Democrat’s margin in any one of the races, his model assumes that’d he’s likely to be off in the same direction in the other Senate races as well. Your calculation of an expected number of errors of 2.46 assumes independence across Senate races, so you’ve actually understated the number of errors that Nate Silver expects his model to make.”
This is a great comment—wrong, but great! The beauty of expected value is that it actually doesn’t require the variables to be independent; the expected number of missed calls is the sum of the expected number of missed calls in each state, whether or not those outcomes are correlated.
Now, if I wanted to know how much Nate Silver expects to be off by when he predicts how many states he’ll get wrong? For that I’d need to pay attention to his model of correlation between the different races.
Correction, Nov. 3, 2014: This article originally misstated that the Kansas Senate race is a “three-way contest.” The Democratic candidate, Chad Taylor, dropped out in September, leaving it as a race between Republican Pat Roberts and Independent Greg Orman. (Return.)
