Ever since Donald Trump locked up the Republican nomination, there has been no shortage of recrimination and mockery directed at political analysts who failed to take his candidacy seriously. No one has taken a bigger beating than FiveThirtyEight editor-in-chief Nate Silver, the famed data journalist whose careful reading of poll numbers in the runup to the previous two presidential elections established him as a lonely voice of rationality and wisdom in a sea of dumb, emotional pundits.
Silver did not perform as well during the 2016 GOP primary. As Trump dominated polls last summer and into the fall, Silver and the entire FiveThirtyEight team made it clear that they would not be fooled by the reality star’s supposed front-runner status. At one point, Silver said he gave Trump a 2 percent chance to win; elsewhere, he put it at 5 percent. At an event in New York back in September, he told a live audience to “calm down” about the prospect of Trump’s polls translating into electoral victory.
At the end of January, I examined Silver’s thought process in an attempt to figure out what he missed. Four months later, with Trump as the presumptive nominee and Silver himself engaged in a process of self-examination, it feels appropriate to ask whether the mistakes he made were avoidable—and whether they were mistakes at all.
Is it possible that Silver was correct that Trump’s nomination was extremely unlikely? If so, how should we assess his forecast in retrospect, and what should we expect from him as the general election approaches? I recently discussed these questions and others with Washington Post reporter Dave Weigel and University of Wisconsin–Madison mathematics professor and Slate contributor Jordan Ellenberg.
Our conversation originally aired as an episode of the Slate Trumpcast. To listen to that podcast, click the player below:
Here is a lightly edited transcript of that conversation.
Leon Neyfakh: To start, I wanted to engage with what I think is the most elegant and simple argument I’ve heard from a number of Nate Silver defenders, who say that Nate did not say a Trump domination was impossible, only that it was unlikely, and the fact of Trump’s victory does not magically make that wrong.
Jordan Ellenberg: I’m definitely supportive of that argument, and I’m strongly in the “leave Nate Silver alone” camp on this one. I think somebody who a year ago or even in January thought it was likely that Trump would be the nominee—I think that would have been an incorrect statement at that time, even though it came out being correct.
If you buy a lottery ticket and you’re sure you’re going to win, and you do win, that doesn’t make it a good decision in retrospect.
Neyfakh: And so, when Nate said Trump had a 5 percent chance to win, you’re saying, well, he did have a 5 percent chance, and it just so happened that reality fell within that 5 percent.
Ellenberg: Right. I mean, improbable things happen a lot, right? I mean, there are a lot more than 20 things you could ask about in this world, and given that there’s a lot of 5 percent chances that are coming through.
Neyfakh: Dave, do you buy that?
Dave Weigel: The logic of the statement is totally there, but that’s not what Nate was doing by saying 5 percent, and that’s also not why there’s so much glee in the media about him being “wrong.” The way that Nate phrased this throughout 2015—and really before that, when FiveThirtyEight was starting—he took this tone that journalism that was not data journalism was inherently stupid and wrong. He had statements in all these interviews and was kind of being goaded into doing it, but I think with a little bit of glee at how the 2012 election had gone, saying that a lot of what columnists did was worthless.
And when he was analyzing Trump and coming up with these very low probabilities that Trump could win, it was always with a tone that the media was doing everyone a disservice by pretending otherwise. The most infamous headline is, “Dear Media, Please Stop Freaking Out About Donald Trump’s Polls.” And if you’re saying people are freaking out, you’re implying that they are ignorant about something that’s crazy—not that it’s unlikely, but it’s just not worth worrying about at all. You wouldn’t tell somebody, “Hey, guy in South Florida, stop freaking out about hurricanes!” You wouldn’t say it—like, there may not be one this year, but it’s something you should worry about.
And I feel like that is why there has been so much glee over him not getting this election right, because the way he got it wrong was so arrogant.
Neyfakh: So, Dave, why did Nate Silver think that the polls showing Donald Trump ahead were better off ignored? What was his reason for discounting them?
Weigel: I actually think his explanations for this—as he’s been humbled into making them—have all been very good and correct. It’s that the Republican base as we understood it, with the limited number of elections we had, was not inclined to give someone like this a nomination. It just hadn’t recently, even in anger swells that we’ve seen since Barack Obama was elected president, it didn’t tend to reward somebody with all the heresies of Trump just because he was oppositional. This was based on the “party decides” thesis and on a lot of other things.
It made sense. I think had he just said it’s unlikely that Trump will win, I don’t think you would have seen quite the same level of mockery. I should say, since Trump started winning, the analysis that he, Harry Enten, and the rest of that team have done of who these voters are has been very good. The problem is, a lot of the media ignore it and just kind of cover Trump as a goofy, point-a-hundred-cameras-at-him phenomenon.
The analysis of who actually proved them wrong, I think, has been good and sort of crow-eating, but the analysis of why it could never happen was silly, and I think he’s explained that enough by now.
Neyfakh: You wrote a piece back in December saying that data journalists—Nate Silver among them—were missing something, that there was this huge story that they were just not taking seriously. So, now that Trump is the presumptive nominee, what do you think he missed? What were his assumptions that turned out to be wrong?
Weigel: That article wasn’t even about me being right or anyone being wrong about who would win the primary. It was just the binary question of: Who will be president? This guy will, this guy won’t, or who will be the nominee?
Ellenberg: Well, it was a very nonbinary question this time, which is part of the problem.
Weigel: Right, but somebody wins in an election, and somebody loses. So, if all you’re interested in is somebody winning the election, then indeed you could ignore a lot of other trends. And I always think of our old colleague, Matt Yglesias, who said this about Romney.
He sort of, with a bit of disdain toward how much political media there is, said that if you just ignored all political coverage for the year up to Romney’s nomination, you would have assumed Romney wins the nomination because the polls said Romney would win. And you could have done the same with Trump. And the point I was making is just that the worship of predictability or prediction and odds and trends was ignoring this insane phenomenon happening across the country called the Trump movement. It would have led you to ignore the Bernie Sanders movement, too.
The formative experience of my reporting life was Ron Paul’s campaigns for president, where somebody who never was going to win—and indeed, did not win—ended up moving the direction of the party and creating what looks minor now because Rand Paul, his son, did so poorly, but really did radicalize elements of the party. Before that, I was kind of a history nerd, and I thought Goldwater was interesting—even though he didn’t become president—because he radicalized the party.
So, I thought that the “Everyone in the media shut up and stop panicking about this” argument made by not just Nate, but much less excusably by a lot of replacement-level pundits who didn’t even bother to talk to voters and just said we should all ignore these trends—they were missing this huge phenomenon that is changing one of our two political parties into something kind of dark and more like the French National Front than the party we’re used to.
Ellenberg: Right, but of course it’s not Nate Silver’s job to tell that story, right? I mean, Silver is not a guy who’s going to talk about the meaning of politics. That’s not what he’s doing. He’s there to talk about what’s likely to happen and what’s not likely to happen. So, I think it’s unfair to blame Nate for missing a story that’s not on his beat.
Weigel: No, and I actually—I come not to bury Nate Silver. Like I said before, he’s eaten a lot of crow, and when there is election data to crunch, FiveThirtyEight is very good. It’s more the people who then spun off that to say, “Why is the media spending all of this time on this Trump phenomenon?” and as a second order on the Bernie phenomenon. It’s more when people used the crutch of, well, “Nate Silver says this,” to say that the media should cover something less or ignore something, I thought that was the mistake.
Neyfakh: But one of the pieces of evidence that Nate used to defend dismissing Trump as a serious candidate or a meaningful movement was that the polls that you might look at as evidence of a movement were unreliable, right? These were preprimary polls, and in the past preprimary polls were not reliable. They didn’t really tell you much about what was going to happen when people actually voted. So, that gets me back to this question of: Can we really say—now that Trump is the presumptive nominee—that Nate Silver’s assumptions were wrong?
Ellenberg: This sort of comes to a deep philosophical point: What do you actually mean when you say there’s a 5 percent chance that something’s going to happen in the future? And if you really drill down into that, it’s rather confusing, right? Because what do you mean when you say there’s a 5 percent chance it’s going to rain tomorrow? It either is or is not, right? After the fact, either it did or it didn’t.
What you actually mean—if you drill down into what a meteorologist is doing—is you say, if you look at days that were like today, where the conditions were similar to what they are right now—and if you’re a meteorologist, you have thousands and thousands of days like that—you say, in how many of them did it rain the next day? And maybe that’s 1 out of 20.
Now, sometimes you’re in a situation where the conditions that currently obtain just don’t really have any comparison. They’re really not very much like these particular conditions that have arisen before. And in a context like elections, where you have many fewer data points than you do for weather, that’s a much greater danger.
Here’s the problem if you’re Nate Silver. If something truly weird is happening that’s kind of unprecedented and doesn’t really have a match in previous elections, and you have this website that you run and there’s a lot of money and a lot of people are looking at it, I think you can’t really say, “OK, this is different, and we have no idea what’s going to happen, and it would be inappropriate to put any numbers on the screen.” As I understand it, he can’t actually do that, right?
Weigel: I don’t think that would be economically advisable, no. But I guess, getting back to what he was saying at the time when he gave that low probability, the “everybody stop freaking out” post, the argument was that these early polls are unreliable—which is sometimes true, in terms of the prejudices voters will have going into the actual election—but then he said, well, this percentage of voters supporting Trump represents only a small percentage of voters who will vote overall and a small percentage of Republican primary voters.
And it was just kind of this way of saying, you didn’t need to worry about a Donald Trump movement because it was a fraction of a fraction of a fraction, based on early polls. That just didn’t strike me as correct. Lots of movements that succeed are not majority movements, they just take over politically because they understand the system and overwhelm in the right areas.
It was a strange argument about why you should not pay attention, because it seems the reader or watcher of TV would say, gosh, it seems like a lot of people are showing up to hear this guy ramble for an hour about trade and Muslims. And the response from FiveThirtyEight was, yeah, but that’s not that many people. That just didn’t seem like a good way to cover the election.
Neyfakh: What would have been the more responsible or mathematically sound way to interpret what Nate was saying? When he says Trump has a 5 percent chance—this goes back to Jordan’s point about the weather—I think for nonstats people like myself, there’s something nonintuitive about that. My brain immediately wants to know, 5 percent of what? What information is contained in that forecast, Jordan?
Ellenberg: Again, I think this is where it gets complicated. Because you could say that it means under 5 percent of similar situations, the guy who was like Trump won 1 out of 20 times, but there probably aren’t enough similar situations for that to really make sense. Probably the best way to think of it is that it’s expressing some kind of degree of belief.
What I like to do is, there’s a site where you can type in the probability and it translates it into baseball terms. In other words, if you type in 5 percent it will be like, OK, a team that is down three runs and batting in the bottom of the eighth wins 5 percent of the time. That’s kind of a calibration for your mind if you follow baseball, to say, that’s how unlikely it is. Like, if your team is down three in the bottom of the eighth, you haven’t given up. You could win. Probably you’re not going to win. That’s kind of a way of thinking about what a 5 percent probability really means.
Neyfakh: I should also note that Nate himself in a podcast the other day talking about this whole thing called back to his quantitative forecast—the 2 percent, the 5 percent. He said: These weren’t the results of a model that I built; these were subjective odds that I gave. They were kind of arbitrary. That was a quote, “kind of arbitrary.”
When I heard that I thought to myself, These were predictions expressed as numbers that I interpreted as statistical, in some basic way. But I have a hard time understanding how that’s not just punditry. Dave, what do you think?
Weigel: I think that question is really only fully answerable by Nate himself, but it certainly sounds like it was just punditry. It was punditry using the chits that he had from things that were not pure punditry, right? He might have felt in his gut four years ago that Florida was more of a Romney state than he predicted, but the data said this and so he went with that.
I think it was much more punditry, and I think he’s said as much several times. He said this after the British election and the Scottish referendum, where both times he was called to analyze something he honestly didn’t know that well. He was looking at that data for the first time, and he apologized for being a pundit, which we know from the rest of his writing is the lowest thing you can be. It’s something kind of—it’s like the crustacean, something way down the food chain. Pundits are terrible.
I think Nate was compelled to make those early predictions about Trump probably for the reason a lot of people who know less about data wanted to underrate Trump. I mean, this was not something I really indulged in, although I think if you had run the election with the same facts a hundred times and in some of those cases Marco Rubio had not muffed it at the New Hampshire debate, it might have been a different primary. I think you still would have had that level of support for Trump, but it might have ended a little bit differently. Who knows?
And again, if you were to line up all of the people in a tribunal who got this wrong, I think Nate was not the worst at all. He always caveated, and he always had reasons apart from that one “media, shut up, stop freaking out” column. I think the Marvel No-Prize has to go to Nick Kristof, who just imagined a conversation with a Trump voter without talking to one. There are more than 10 million of these people already. I don’t think Nick Kristof …
Ellenberg: He didn’t even find a cab driver. Aren’t you supposed to find a cab driver who’s voting for Trump and talk to that guy?
Weigel: I don’t think Nick Kristof would put up with me if I said, “Let me imagine a conversation with a Syrian refugee. What’s he like? Let me just guess.” There was a lot of that that was much more harmful.
Ellenberg: Or what if Nate just started imagining taking polls. I imagined there was a poll. There wasn’t one, but you know, it would be good if there was.
Weigel: Imagine a housewife who answers a poll question. Imagine the pollster. No, I think he was indulging in punditry there, and I’m just trying to back up and defend him a little bit in saying, OK, well, at worst it just didn’t have this patina of science behind it, right? People wanted to believe there was some reason they could ignore Trump or that Trump couldn’t happen, and so they said, “Well, Nate Silver says it.” And no one else had that power. So, I think, like Spider-Man failing to save Uncle Ben, he’s using these powers he’s been given for something less highfalutin than he could have.
Neyfakh: Right, but he said 5 percent. That’s a number. And when you hear Nate Silver—the guy who is the king of numbers—tell you that something is 5 percent likely, you’re going to interpret that as a formally derived forecast that came from some model. Was it irresponsible for him to use numbers in that way, to sort of dress up what it sounds like we all can agree was roughly punditry in the clothing of statistics?
Ellenberg: This is where I feel like it would be super hard to be Nate Silver. I don’t know the guy that well, but I sort of think of him as somebody like me. I put myself in his shoes, and I feel like I’m sure he doesn’t really want to use hard numbers at a time when nobody’s voted and you have these preprimary polls. That’s a low-data environment. And yeah, I think in that context there’s no choice but to have a lot of guesses built into your model. Even if it’s formal, there are a lot of guesses in it.
The closer it gets to the election, the more he’s going to be able to say, “This number is meaningful. It comes from something formal.” But I guess I just don’t think he has the option to wait.
Neyfakh: Because he has a website.
Ellenberg: Right.
Neyfakh: In the past, Nate and FiveThirtyEight have been really, really good at forecasting general elections—I believe they got every state in 2012. But they were very bad at forecasting Trump in this election, at least sort of in the macro sense. How should we feel about the site and about Nate and about his methods, as they set about forecasting a Trump general election?
Ellenberg: I think it’s a huge net positive to have this in conjunction with the existing infrastructure of punditry. I think having this kind of methodological diversity is a massive net positive. I think that every single model is based on the idea that conditions now are in some way similar to conditions that have obtained in the past. If that’s not true, then I think we should be a lot less confident.
Actually, I have a question for Dave: Do you think there’s going to be a big “shy Trump-y” effect? Are there going to be, systematically, people lying to polls and saying they’re not going to vote for Trump when actually they are?
Weigel: Well, I want that to be studied, because I feel some of that might have already happened in Maryland and Pennsylvania. Maryland is a good place to look at that example, because George Wallace, when he ran in the 1964 primary, did much better than anyone expected there. The same thing with Wisconsin.
Trump has outperformed when the electorate was Republicans and ornery ex-Democrats. That’s something that I’d want people with the brain power of Nate Silver et al to wrestle with before we get to the election. I remember this coming up with Barack Obama in 2008. I just saw no evidence based on recent elections with black candidates that there was a “Bradley effect,” that people would tell a pollster they were voting for the black guy and not do it. I didn’t see that.
But the combination of toxic Trump unfavorables and slightly less toxic, maybe just food poisoning–inducing Hillary unfavorables, I think does create this risk there. I should say, Democrats have their own theory, which is that there will be people who tell maybe even their husbands they are going to vote for Trump and then they will come out and vote for Hillary Clinton. So, this is a general election that still very much needs actual data.
Ellenberg: The point is that if there are large-scale effects like that, I think they’re—Dave, you would know this better than me—but I don’t think we’ve had massive misstatements to pollsters in previous general elections. If that kind of effect is in place in this election, then all bets are off, right?
Weigel: No, we often have misinterpreted—by “we,” I mean people who are not us—have misinterpreted polls that were kind of wrong about the election as, maybe people lie to pollsters. And what, for example, the victorious Obama campaign in 2012 would say was, no, our data internally said we were going to win by this much. Romney’s didn’t, because we modeled better than they did and we knew where people were coming out.
And that’s going to be another tricky thing about this election, because the Hillary campaign has always bet on just grinding this out by getting every single possible Hillary voter to the polls and knowing who they are and what their pets eat for breakfast and everything. And the Trump campaign is going to be, “I’m Trump. Here’s a big rally. Go out and vote for me.” So, it’s going to be really hard—harder—to predict on that basis from the Republican side.
Ellenberg: I agree. It’s going to be harder to figure out based on polls than usual, but I’m just being a pundit when I say that, not a mathematician!
Read more of Slate’s election coverage.
