Victory Lab

“Likely Voters” Lie

Why private campaign polls get such different results from public media polls.

Voters at a polling station.
A new paper suggests that pollsters asking voters if they plan to vote is no better than flipping a coin

Photograph by Digital Vision/Thinkstock.

After the 2008 election, the Democratic polling firm Greenberg Quinlan Rosner collected the names of 12,979 people who had, over the course of the year, described themselves to call-center operators as “almost certain to vote” or “will probably vote” and checked to see if they actually had. Masa Aida, a Greenberg analyst, matched respondents’ names to a voter file maintained by the database vendor Catalist, and found that citizens had an unimpressive record of predicting their own behavior: Eighty-seven percent of those who described themselves as “almost certain to vote” that November had done so, compared to 74 percent of those who said they “probably” would.

This wasn’t much of a surprise. Political scientists have found for years that citizens, aware of the socially correct answer, routinely lie to claim they voted when they had not. But Aida, along with Harvard public policy professor Todd Rogers, did something that past researchers hadn’t. They also looked up the records of those who had said they “will not vote,” an answer that prompts the operator to politely end the call and dial someone else. Greenberg Quinlan had excluded those people from their surveys, but Aida and Rogers found they were lying too, and at a higher rate than those who identified themselves as certain voters. Despite claiming they would not cast a ballot, 55 percent had. More than half the people whom Greenberg Quinlan call-center operators kicked off the line should not have been.

That “likely-voter screen,” the battery of questions (Gallup’s contains seven of them) asking people how probable they are to vote, is a staple of just about every survey you see these days. “Public polls”—the ones that now pop up with metronomic regularity under the banner of news organizations and colleges—typically rely on such screens to filter likely voters from the much broader pool of people they get on the line by randomly dialing numbers. In their unpublished paper, Aida and Rogers poked big holes in the screen, suggesting in some cases it was no better than flipping a coin in determining who was likely to vote.

This is one reason why the likely-voter screen is becoming an afterthought in the parallel world of private polls—the ones commissioned from partisan firms to guide strategy, and rarely seen outside campaign war rooms. In fact, a large methodological gap has now opened between the surveys candidates and their strategists see and the ones you do. Campaigns are, in essence, relying less on voters’ honesty and self-awareness to determine who is a likely voter. Instead, they’re using their own statistical work to presume who is likely to vote, and surveying based on those statistical guesses.

“I certainly have more faith in even a Democratic poll than a media poll,” says Jon McHenry, a Republican pollster whose firm, Ayres McHenry, is working for the Our Destiny super PAC backing Jon Huntsman. “I trust that the Democratic firm is doing it the same way our firm is doing it.”

The difference between the public and private approaches has particular consequences in Iowa, where the record of who has turned out for past caucuses is a closely held secret unavailable to most media or academic pollsters. But there’s another reason for the public-private gap, which news organizations rarely mention when they put out new numbers and frantically rearrange daily coverage around them: Many of their polling strategies are not developed with the foremost goal of assessing the horse race. Instead they want stable datasets that allow them to track public-opinion trends on noncampaign issues, such as abortion or presidential approval, even in election off-years. 

“The main thing is we want to know what everybody thinks, not just voters,” says Washington Post polling manager Peyton Craighill, who worked for Greenberg Quinlan before making the jump to media. “The political pollsters all have their own special sauce.”

So how do campaign pollsters operate differently? Instead of randomly dialing digits in a given area code and then imposing a screen to sift out nonvoters—the public polling strategy—campaign pollsters usually begin with a list of registered voters available from state election authorities. Those lists include individual voting histories, which campaigns have always used to individually identify their targets. It makes sense in a primary to send a canvasser to first knock on the door of those who regularly vote in primaries, or give a phone-bank volunteer a list of those who had voted in two of the last four elections rather than a registered voter who hadn’t turned out during that time.

Campaign pollsters usually purchase these lists from vendors who have compiled the local voter lists into national databases, then merged voter names with telephone numbers. (Catalist, the dominant Democratic data vendor, claims it has phone numbers for 88 percent of active registered voters.) Campaigns then feed what they learn from doorstep visits and phone-bank callers back into the databases, where it supplements individual demographic information. Over the last decade, campaigns have become deft at using microtargeting algorithms to analyze all that information to produce a unique score predicting how likely someone is to vote, a determination that is more nuanced and dynamic than just counting how many of the last four elections they had attended.

Working from such lists, campaign pollsters are able to define the universe of potential interviewees, ruling out any chance of randomly dialing noncitizens, the unregistered, or wrong numbers.  But it’s about more than cutting call-center costs.  Campaigns already rely on voter files, and the scores that microtargeting algorithms pull from them, to set their vote goals. Now by pulling their polling samples from the same pool of data they use to count votes, strategists have synchronized their assumptions about who is likely to turn out across different parts of the campaign.

The private poll filters are far more rigorous than anything public pollsters do. “We’re not experts in turnout,” says Des Moines-based pollster Ann Selzer, whose Iowa Poll for the Des Moines Register is widely regarded as the most reliable of the state’s surveys. For general elections, Selzer randomly dials Iowa numbers and asks to speak to a registered voter. During caucus seasons, she calls from a list of voters made available by the secretary of state, and discriminates by party. Now, she’s randomly dialing those registered as Republicans or independents to “get rid of any Democrats right off the bat.” Once she reaches a particular voter, Selzer relies on a screen to filter out her unlikely voters.

Given the different methods, public polls are likely more volatile than those being used by campaigns to guide strategy and tactics, since they rely only on a momentary declaration of interest. After all, an infrequent voter who gets ginned up by a Rick Perry ad right before the pollster calls will get past a likely-voter screen, while a campaign surveying past caucus-goers would never ring her in the first place. In a race like this one, where Republican primary voters have sent mixed messages about their enthusiasm for the choice they face, internal polls will sample a more stable electorate. Those candidates, like Ron Paul, who aim to enlist new caucus-goers probably look stronger in media polls than those commissioned by campaigns—perhaps one reason his rivals might not take him so seriously.

Indeed, campaigns have stumbled in part because their insular polls have offered an insular view of the electorate. Selzer noted that in 2008 Hillary Clinton’s campaign predicted Iowa turnout based on past patterns, using costly rosters of prior caucus attendees controlled by the state party. Many of those who caucused for Barack Obama would not even have appeared in voter databases or caucus lists because they were not previously registered. “When anyone tells you they have a turnout model, you should be suspicious,” Selzer says. “The best predictor of the future is past behavior—until there’s change.” (Even Obama’s campaign dramatically underestimated turnout in 2008.) 

In fact, when they checked Greenberg Quinlan’s self-identified likely voters against their eventual 2008 turnout, Aida and Rogers found that people most accurately described their future behavior when the prediction matched what they had done in the past. Among respondents who had voted in both of the previous two elections, 93 percent of those who said they would vote did so; only 24 percent of those who said they would not vote actually failed to vote. (A similar pattern held among those who had not voted in the past two elections.)

One possible reason that regular voters might consistently declare their lack of interest in voting, Aida and Rogers speculate, is “to convey disaffection toward the political process rather than a sincere lack of intention to vote.” The question of whether it’s better to include such people in a poll or just leave them out altogether remains open. “If I can’t trust them to be honest about whether they’re going to vote or not,” asks McHenry, “how can I trust them on all the other questions I want to ask them?”