Hey, data journalists: Lay off the Scottish referendum pollsters.

Hey, Data Journalists: Lay Off the Scottish Referendum Pollsters

Hey, Data Journalists: Lay Off the Scottish Referendum Pollsters

Reporting on Politics and Policy.
Sept. 19 2014 11:15 AM

Hey, Data Journalists: Lay Off the Scottish Referendum Pollsters

Don't trust the psephologists.

Astrid Stawiarz/Getty Images

The final average of polls in Scotland's independence referendum pointed at a narrow 52-48 victory for the "No" campaign. (Fine, fine: The "Better Together" campaign, if you want to be a stickler.) The final result was 55-45 for "no." The gurus at UK Polling Report concede that this looks like "some systemic error," and that a final YouGov survey might have revealed that one in 50 voters ended up changing their minds as they closed the door on the polling booth. YouGov's final "recontact" showed a 54-46 break for "No," close enough to the result.

Yet this cannot save the pollsters from a Nate Silver post titled "The Scotland Independence Polls Were Pretty Bad." Silver admits that he "behaved like a television pundit" last year when he was asked about the vote by a Scottish audience if "No" had a chance. "If you look at the polls," he said then, "it's pretty definitive really where the no side is at 60-55 percent and the yes side is about 40 or so."


And it was definitive. In all of 2013, the only poll that had "Yes" cracking 40 percent was conducted by the Scottish National Party. For much of 2014, the "No" lead was steady, with the occasional close result spurring a few wouldn't-it-be-funny-if news stories, then fading. It stayed like this until the last two weeks of the campaign, when a debate win by the SNP's Alex Salmond over Better Together's avatar Alistair Darling preceded a "Yes" surge.*

Now, it's true that if you locked yourself in cryo-freeze nine months ago and expected a "No" win, you'd have been right. Just as you'd have been right if you said "I bet the frontrunner, Mitt Romney, wins the GOP nomination" in 2011, and ignored everything else that happened. If you want to shake your head at the reporters who wrote that "to walk through virtually any Scottish town this week was to be confronted by an apparently unassailable army of people voting Yes," you can!

But you would have missed a panic that changed the future of 5.4 million Scots. The "No" coalition, composed of the three main British parties, responded to the "Yes" surge by pushing up or amending their own promises for further Scottish devolution. They had been expected to offer some of this stuff in six-odd months, in party manifestos. Instead, they tried to head off a "Yes" win by assuring wavering voters that unemployment programs, some housing programs, and significant taxing powers be devolved to the Scottish parliament. Banks and investors threatened to bolt the country. Ironically, this decision of "yes" or "no" was less binary than the standard U.S. presidential election. When one of our candidates wins, he wins. When the "Yes" campaign got close to winning, the "No" campaign made huge concessions, and this simply wouldn't have happened had the polls stayed at 60/40 for "No."

Silver points out that a YouGov poll that showed "Yes" surging "did so only because of its weighting procedures" and "on an unweighted basis, it had 'no' ahead by about 6 percentage points." That's an imporant point, being lost elsewhere. But what does it tell us about how "No" should have responded? Should they have rested comfortably in the knowledge that the "Yes" surge was slowing? Should they have promised less to Scots? Even if that unweighted poll had been dead-on, they'd have seen the union saved by a 53-47 margin, down from the 20-point margin of a few months previous.

*Correction, Sept. 19, 2014: This post originally misspelled Alistair Darling's first name.

David Weigel is a reporter for the Washington Post.