HOME /  Frame Game :  How you look at things.

Pollish Sausage 

No matter who you think is going to win the presidential election, you can find a poll to back up your opinion. If you're betting on George W. Bush, you can point to the Voter.com Battleground 2000 survey, which consistently shows Bush ahead. If you're betting on Al Gore, you can point to the New York Times/CBS poll, which usually indicates a small lead for Gore. If you think the debates helped Bush a lot, you can point to the CNN/USA Today/Gallup poll, which found a big Bush surge after each encounter. If you think the debates didn't help Bush much, you can point to the Reuters/MSNBC/Zogby survey, which has rarely shifted more than two points a day.

Advertisement

Why do the polls confirm so many theories? Because theories are built into the polls. Each polling outfit has its own objectives and biases. In the case of media surveys, these objectives and biases aren't about ideology; they're about news-making and social science. Some tracking pollsters want to find big day-to-day changes, others want stability. Some want to narrow the population they study, others want to broaden it. Some fear passive bias, others fear active bias. Each pollster designs his survey to suit his preferences, and each gets the results he's looking for. Like the rest of us, pollsters have theories about who will vote and how. Polls don't confirm these theories. They incorporate them.

This year's big controversy is the CNN/USA Today/Gallup tracking poll. Other pollsters are dismayed at Gallup's radical swings. In the two days after the first debate, Gallup's three-day sample went from an 11-percentage-point Gore lead to a seven-point Bush lead. Last weekend, Bush had a nine-point lead in the Gallup sample; two days later, Gore had grabbed the lead. Contrast this with the Zogby survey, which moved only four points and two points during those periods, respectively. Why the difference? Because Gallup and Zogby are looking for different things. Gallup is trying to capture daily fluctuations, while Zogby is trying to filter them out.

On its Web site, Gallup makes clear that its poll seeks to maximize daily change: "Our objective is to pick up movements up and down in reaction to the day-to-day events of the campaign." Gallup postulates that one in five voters is highly malleable: "A sizeable portion of the voting population, upwards of 20%, is uncommitted and on any given day as likely to come down in favor of one candidate as the other." Gallup doesn't mind that big shifts in the partisan makeup of each day's sample—one day lots of Republicans, the next day lots of Democrats—push its numbers back and forth. Gallup's editor in chief, Frank Newport, says these partisan shifts reflect "differential intensity" between the parties. One day, Republicans feel likely to vote; the next, Democrats feel likely to vote. Accordingly, the pool of "likely voters" shifts from Bush to Gore.

Other pollsters regard that kind of change as a distraction. They want to hold some factors constant—including party affiliation—so they can focus on variations in other factors. "We're trying to measure movement within groups," says Ed Goeas, the Republican pollster who oversees the Voter.com survey. "If I see that white women have moved 10 points, I want to see whether that was real movement"—as opposed to an excess of Republican women in the first sample and an excess of Democratic women in the second. Similarly, Washington Post survey director Rich Morin writes that Gallup "may not be tracking real changes in the electorate, but merely changes in relative interest or enthusiasm of Republicans and Democrats."

Notice the clash of premises. Morin and Goeas use a hard model of voting behavior. They assume that any changes in the horse-race numbers (i.e., the percentage of respondents who plan to vote for Bush or Gore) caused by changes in the partisan makeup of the likely voter pool aren't "real." These pollsters treat the distribution of Democratic and Republican voters in presidential election turnout as a constant. When they see poll results in which that distribution shifts back and forth like a variable, they dismiss the data and fault the poll's methods. You could argue that their hard model, with its fixed dichotomy of constants and variables, is too rigid. But you could argue just as easily that Gallup's soft model, which treats everything as a variable—to the point of positing that uncommitted voters are "on any given day as likely to come down in favor of one candidate as the other"—is too mushy and chaotic. Which model is better? The answer to that question isn't scientific. It's philosophical.

It's also practical. CNN and USA Today are in the news business. They're paying Gallup for new numbers every day. If Gallup's numbers don't change, where's the news? So Gallup has an incentive to keep its filter loose, allowing the winds of shifting partisan intensity to blow its numbers back and forth. Goeas, on the other hand, is a professional campaign pollster—as is his Democratic partner in the Voter.com survey, Celinda Lake. They've designed their poll to get the kind of information a candidate, as opposed to a news organization, would want. Campaigns divide the electorate into demographic groups—union households, white women, Midwestern Catholics—and target their ads and messages to those groups. A campaign manager needs to hold the distribution of these groups constant from day to day so she can track movement within each group. Which poll is correct? That depends on what you need the numbers for.

Here's another philosophical question: How many days do you need to poll in order to understand public opinion? Gallup is sampling 400 people every night. Since CNN and USA Today want the numbers to keep changing, they report a rolling average based on only the last three samples. If Gore was doing well three nights ago, but Bush is doing well tonight, the pro-Gore sample drops out of the three-night mix, the pro-Bush sample goes in, and Bush gets a big bump. On its Web site, however, Gallup reports a rolling average based on the last six samples. The pro-Gore sample stays in the mix, diluting Bush's bump—and conversely, tonight's pro-Bush sample stays in the mix five days from now, diluting Gore's next bump. The result is a less exciting series of smaller shifts. The three-day average tells you how 1,200 people feel right now. The six-day average tells you how 2,400 people feel over the course of a week. Which number should you pay attention to? That depends on whether you want the latest news or the big picture.

The argument for the big picture is that it's a better predictor. Presidential preference "is not a firmly held attitude," says Gallup's Web site. "[T]here is no need for Americans to develop a firmly held view on their vote until Nov. 7." Yet Gallup says its poll is designed to clarify who would win the election if it were held today. Its surge toward Bush after the first debate, for example, suggests that "if the election were indeed held during the days after the debate, Bush would have won, in large part because his voters would be more likely to turn out to vote." But if presidential preferences don't become "firmly held" until Election Day, then it makes no sense to infer from today's numbers that Bush would win "if the election were held today." The election isn't being held today—and if it were, voters would have to resolve their fluctuating feelings into firmly held views that might not lead to the same conclusion.

Every pollster dreads statistical bias. But there are two kinds of statistical bias: passive and active. Passive bias is what happens when you don't balance your sample. If you live in a white neighborhood and poll your neighbors, you don't get enough black respondents. You have to take steps to make sure you either 1) sample the proper percentage of blacks up front; or 2) "weight" the number of blacks in your sample to reach the proper percentage. For example, if you polled half as many blacks as you should have, you double the weight of each black respondent's answers, as though you had polled the correct number.

SINGLE PAGE
Page: 1 | 2
MYSLATE
MySlate is a new tool that you track your favorite parts Slate. You can follow authors and sections, track comment threads you're interested in, and more.

Will Saletan covers science, technology, and politics for Slate and says a lot things that get him in trouble.