Victory Lab

Knock, Knock

In Mexico, polling is done the old-fashioned way: In person. Could this be the future of polling in post-land-line America?

The presidential candidate for Mexico's Institutional Revolutionary Party (PRI), Enrique Peña Nieto, waves at supporters during an electoral rally in Toluca, Mexico State, on June 27, 2012.
The presidential candidate for Mexico’s Institutional Revolutionary Party (PRI), Enrique Peña Nieto, waves at supporters during an electoral rally in Toluca on June 27. Sunday.

Photo by Mario Vazquez De La Torre/AFP/Getty Images.

MEXICO CITY—One Saturday last July, in the central Mexican state of Michoacan, where the country’s first constitution was written, two cars carrying six visitors to the city of Apatzingán disappeared. Three days later, in nearby La Cofadria, another three out-of-towners vanished. The circumstances were not mysterious—Apatzingán is known as a base for drug cartels and has been a site of deadly violence—but the targets of the kidnapping were. The nine hostages were all survey-takers who had come to Michoacan to poll residents’ preferences in upcoming gubernatorial elections and were working for firms commissioned by the leftist Democratic Revolution Party. For Mexico’s well-developed polling industry, days of waiting for signs of life invited some self-examination. Did someone consider their research a threat? “Suddenly the cartels see a lot of cars with strange plates and ask: who are these people?” Ulises Beltrán, the country’s most venerable political pollster, speculated the other evening. “Are they the police? Are they the FBI?”

None of the nine who were seized—and released days later—worked for Beltrán’s firm; they had been employed by two rivals, Consulta Mitofsky and Parametria. But Beltrán had last summer’s suspenseful week on his mind because the incident marked perhaps the greatest threat to the polling method he had helped to pioneer in Mexico. On Tuesday night, Beltrán was in his office in Mexico City’s La Condesa neighborhood, putting the finishing touches on a poll for the next day’s edition of the newspaper Excelsior. As is the case with polling in much of the world, this required putting boots on the ground. Because it was the last of the polls that would appear before this Sunday’s presidential election—Mexican law prohibits the publication of polls over the final four days before voting—Beltrán had increased his sample size. Now, instead of interviewers canvassing 120 of Mexico’s electoral precincts to speak to 10 voters in each, they had been scattered across 150. That meant one-quarter fewer logistical worries than usual. 

American pollsters once saw such tactile survey methods as a primitive practice to be used only in countries yet to graduate to surveys by telephone. But many are adjusting to the reality that a return to in-person interviews may be one of several imperfect solutions to their own crisis: a rapidly growing inability to reach voters by phone. “It’s a lot harder,” says John Anzalone, who has polled for 20 years on behalf of Democratic candidates, including Barack Obama. “It used to be that basically everyone had a land line—it’s how people communicated. Other than Dad saying, ‘You can’t answer the phone during dinner,’ people liked answering the phone. You had a captive audience.” Visiting voters at home is expensive, time-consuming, and occasionally fraught with real peril, but as American political polling reaches the end of its High Modernist era, practitioners are desperate simply to get back in front of people. “It’s a throwback,” says Anzalone. “But you’re crazy not to be experimenting with a new way of capturing them.”

***

In 1988, the newly elected Carlos Salinas de Gortari hired Ulises Beltrán to be the first presidential pollster in Mexico’s history. Election polling was effectively nonexistent in the country, a reflection of the fact that there was rarely a horse race worth tracking: At that point, Salinas’s PRI had presided over unchallenged one-party rule for more than 50 years. But the new president wanted to take stock of attitudes about policy questions and asked Beltrán—who had studied economic history at the University of Chicago and was then working as a statistician in Mexico’s version of the IRS—to design a method of measuring national public opinion.


Even though American political consultants were working across Latin America at the time, there was little that Mexicans could learn from them about basic survey practices. American pollsters built their samples by randomly dialing phone numbers and then weighting the results if the sample they collected failed to represent the demographics of the broader population or electorate. But fewer than one-tenth of Mexicans owned a telephone line, and the ones who did were far from representative of the country: They were wealthier, better-educated, more urban. Beltrán knew he would have to conduct all of his interviews in person.

A half-century earlier, American pollsters had been in a similar position. Beginning in the 1930s, George Gallup sent interviewers to hundreds of locations nationwide and asked them to randomly select individuals in certain demographic categories for questioning. This method was seen as a (far more expensive) corrective to the straw polls that Literary Digest had conducted since the 1920s, by sending millions of ballots to mailing lists built off its subscriber records, phone books, and automobile registries. Gallup’s method was validated in 1936, when he accurately predicted Franklin Delano Roosevelt would be re-elected; Literary Digest’s poll showed Alf Landon winning 55 percent of the vote. But it was another polling flub 12 years later—the “Dewey Defeats Truman” prognostication—that forced survey-takers to be more rigorous about their methods of selecting households, a shift that dramatically increased the costs of polling. Instead of sending his survey-takers to public places, Gallup began dispatching them to specific doorsteps.  

Beltrán developed a similar approach. After Mexico completed its 1990 census, he hired its newly unemployed researchers and turned them into an army of political interviewers. He randomly selected electoral precincts and then within the precinct a block; interviewers started on one side and knocked every third door. If no one was home, interviewers would return later. “It depends how much money you have,” says Beltrán. “If you have a lot of money and time, you go back until you get the person you want.”

By then, American survey-takers had moved indoors. In 1970, as the share of Americans with telephone lines crossed 90 percent, pollsters realized they could run all their surveys from call centers. By randomly dialing digits they could have greater confidence that they were getting a statistically valid representation of the phone-owning population. When interstate telephone service was deregulated, the cost of polling by phone dropped even further. The only researchers who stuck to knocking on doors were those like the Census and American National Election Studies, which had long questionnaires, deep budgets, and a goal of covering the entire population. For everyone else, polling—especially with the introduction of automated touchtone surveys—became something that could be done cheaply, widely, and often.

As Mexico’s 1994 elections approached, Beltrán worked for the campaign of Salinas’ successor Ernesto Zedillo. The PRI was finally facing real electoral challenges, and knocking on someone’s door and asking how he or she planned to vote seemed a potentially more complicated intrusion than merely asking for one’s views of economic conditions. Beltrán ran a battery of randomized experiments to test whether results would be biased by particular aspects of the interaction. Did the questioner’s gender alter the interview dynamic? Did the color of the pen guide the respondent to one party or the other? Beltrán found they had little impact, and those factors that did likely cancelled each other out across a large sample. “In the big numbers,” he says, “cultural effects disappear.”

The greatest concern, however, was one to which the PRI was particularly sensitive: a fear that citizens, suspecting that the clipboard-equipped visitor at their door could be a government agent, would refuse to answer questions—or exaggerate their support for the incumbent party. To mimic the way Mexicans cast ballots on election day, Beltrán sent his interviewers into the field with paper ballots and a box. When it came time to ask which candidate they would support if the election were held that day, voters would be handed a facsimile of the official ballot (which included the party’s logos in full color, so illiterate voters could distinguish them) and asked to put an X on their choice and place it into the box. “It is the only way to do it, elegantly,” Beltrán says.

In 2000, when the PRI was voted out of the presidency, Beltrán left government and started his own firm. His team of former census-takers set out to become their own company, one of several contracted by pollsters to conduct their fieldwork.  Now Mexico has around 10 top-flight pollsters, most of whom have a contract to run surveys for a newspaper and some of whom simultaneously work for political parties and other private clients. Even as the share of mobile-phone users increases, there is still fewer than one land line for every five Mexicans. Pollsters may use phones for tracking polls, designed to discern movement. Nearly every serious poll designed to measure the views of the electorate continues to rely on face-to-face interviews, even though pollsters estimate they are twice as expensive as running the same surveys by phone. “Most reliable pollsters do at-home polling,” says Javier Aparicio, a political scientist at Mexico’s CIDE research institute who is currently serving on the national body tasked with administering Mexico’s election. “Some pollsters do phone-only, but they’re not well-regarded because they do it on the cheap.”

As Mexico has urbanized, more voters live in well-mapped areas with clearly defined blocks that are easy to randomize. In rural areas without such orderly geography, Google Maps and satellite imagery allow researchers to identify clusters of homes that can be treated as blocklike units. There, pollsters still have to allocate extra time for interviewers to get from door to door, and often to call in an interpreter when they stumble upon a voter who speaks only an indigenous language. Even in urban areas, challenges persist: It remains very difficult to find men between ages 30 and 39 at their homes at any time other than the weekend.

But it is the physical safety of interviewers in the Mexican states most directly affected by the drug war that has posed the greatest complication. Jorge Buendía, whose firm Buendía y Laredo polls for El Universal newspaper along with political clients, says that each time he randomly pulls 150 electoral precincts to survey, he expects to discard two or three because they are in parts of the country where it is simply too dangerous to dispatch interviewers anymore. He has also learned not to ask directly about the local security situation, because respondents are usually too intimidated to answer, and tries to keep interviews to 30 minutes or fewer, in part because they are typically conducted while standing at the doorstep. Interviewers are advised not to enter respondents’ homes “for the safety of both sides,” according to Buendía.

Since last summer’s Michoacan kidnapping, Buendía has made a point of ensuring his interviewers go out into the field with even less protection. Instead of teams, they travel alone, and without handheld devices or laptops. It sounds counterintuitive, but it’s strategic. “In groups, they draw the attention of drug dealers,” Buendía said. His pollsters now set out with little more than clipboards, questionnaires, and the Beltrán innovation known as the “simulated voting box,” to re-create the election-day experience on a doorstep. “The basics,” he says, “have more or less stayed the same.”

***

In the United States, however, the polling industry faces a reckoning. Last month, the Pew Research Center, in what it called a report but was best read as a plaintive appeal for sympathy, declared, “It has become increasingly difficult to contact potential respondents and to persuade them to participate.” In 1997, Pew reported, the telephone response rate to its polls (the share of outbound calls that yielded an interview) was 36 percent; by 2012, it had fallen to 9 percent. It was easy to attribute blame—the proliferation of mobile devices, caller ID, and the increasingly secondary nature of phone communication altogether.

There are alternatives to traditional phone polling, but all of them have trade-offs. The changing landscape is not only making polling more arduous and expensive but has also cleaved American society into demographically (and often politically) distinct groups, much as Literary Digest did when it polled only magazine subscribers and car owners during the Great Depression. In 2008, the Obama campaign was jarred to find that it was running even with John McCain among young voters it polled via traditional land lines. When campaign officials commissioned a separate poll using only mobile numbers—a population more likely to include minorities and students—Obama’s expected lead over McCain fully materialized.

But just dialing mobile phones is an uneven endeavor. While many voter files now include mobile numbers, and others are available from commercial data vendors, laws restrict the use of some automated dialers when ringing cell phones. Some researchers are using the Internet to conduct surveys, but traditional sampling there is impossible. Those designing Web surveys are forced to either use offline contact to assemble an online sample (by calling phone numbers and recruiting people to join a Web panel) or altogether forgo the ambition of creating a random sample and rely on other methods to ensure that respondents are representative of a desired population. 

Knocking on doors would likely be the most methodologically pure of the post-phone approaches, although also the most demanding. Campaigns might find it a worthwhile addition to their research toolbox, especially for so-called benchmark polls that offer an early, expansive survey of the electorate to guide strategy and planning. But many would likely find it hard to justify the practice. “In-person interviewing is a nonstarter for media polls. It’s too expensive—most newspapers have cut their budgets for polling and can barely afford phone surveys—and takes too long,” says Douglas Rivers, a Stanford political scientist who has pioneered online surveys and is currently chief innovation officer for the pollster YouGov. “The prospects for door-to-door interviewers in the 21st century are about the same as for milk men.”

Polling may be less like dairy than energy reserves. American public-opinion research has likely reached its version of peak oil, with the cheap, plentiful polling that undergirded political communication for 75 years no longer sustainable.  Quick surveys may no longer be within reach of every blog, trade association, liberal-arts college, and state-legislative candidate. As pollsters try new techniques for reaching people, their choice of methodology will betray divergent priorities: cost vs. speed vs. representativeness. It will be harder to pretend all polls ought to be treated equally.