Supporters of President Bush's tax cut plan are trying to convince Congress and the media that the plan adds up. Critics are claiming the opposite. But that isn't the only numbers game driving the debate. Bush's foes are trying to rally congressional resistance to the plan by circulating polls that purportedly show the public doesn't want it. Bush and his allies are trying to break down that resistance with polls suggesting the opposite. Like the budget math game, the public opinion math game is full of skewed and suspect data. Here are a few questions worth asking of each survey.
1. How is the proposal characterized? Last week, the news pages of the Wall Street Journal cited a poll commissioned by the Issues Management Center, a pro-Bush tax cut group, "showing that 57% of South Dakota voters support the Bush tax-cut plan, with 27% opposing it." This came just a week after the Associated Press, trumpeting a Pew Research Center national survey in which the tax cut was favored by just 43 percent to 33 percent, reported that the plan "has found a lukewarm reception."
Why the discrepancy? Look at the questions. The Pew survey asked, "Do you approve or disapprove of George W. Bush's tax proposal?" Since the question didn't mention that the proposal involved cutting taxes, fewer respondents approved. The IMC poll, on the other hand, asked, "Do you support or oppose President Bush's plan to cut taxes by $1.6 trillion over the next ten years?" This time, more people responded favorably—presumably because they liked the size of the tax cut or were comforted to learn that it would be spread out over 10 years.
What happens when you include the basic information omitted by Pew (that it's a tax cut) but omit the additional information included by IMC ($1.6 trillion over 10 years)? You get the question Gallup asked four weeks ago: "Do you favor or oppose the federal income tax cuts President Bush has proposed?" The answer fell halfway between the other polls: 53 percent approved, 30 percent disapproved.
2. What's the comparison? Media-bias watchdogs accuse CBS of squelching data favorable to Bush. Boston Globe columnist Jeff Jacoby, among others, protested that Dan Rather made "no mention of CBS's own poll showing that a whopping 67 percent of the public likes Bush's plan." But the critics made a strategic omission of their own: In the same survey, when voters were asked which of four proposals they preferred—tax cuts, deficit reduction, more education spending, and more money for Social Security and Medicare—only 35 percent chose tax cuts. This is an intuitive law of polling: The more alternatives a proposal is compared to, the more its support shrinks. Bush's allies want a yes/no question that essentially compares his tax cut to nothing. The Democratic National Committee and the Democratic Leadership Council, on the other hand, are peddling surveys in which voters are given a list of several options for allocating the surplus. In these surveys, the percentage preferring tax cuts falls below 20 percent.
A comparative question doesn't necessarily capture public opinion more accurately than a yes/no question does. The 35 percent who preferred tax cuts in the CBS News poll was the biggest of the four factions produced by that question. But by scattering the respondents among those four categories, the question reduced the pro-tax cut figure from 67 to 35. Comparative questions also assume that the alternatives presented are mutually exclusive—an unproven premise that in this case is the crux of the debate. When asked whether it's possible "to preserve programs like Social Security and Medicare, increase spending on education, and implement the $1.6 trillion income tax cuts George W. Bush has proposed at the same time," 60 percent of CBS News respondents said yes. Nevertheless, the comparative question obliged them to assume the contrary.
When a poll poses the question only one way—as the Washington Post/ABC survey did on the eve of Bush's budget speech—readers get only half the story. The Post/ABC poll skipped the yes/no question and instead asked respondents to choose 1) among tax cuts, debt reduction, Social Security, education, and health care; and 2) between Bush's tax cut and a "smaller tax-cut plan that provides targeted tax cuts mainly for lower and middle-income people." In the first question, cutting taxes came in third out of four, with 22 percent support. In the second question, Bush's tax cut trailed 53-43. The poll was widely reported as a rejection of Bush's plan. "Poll Shows Bush Tax Plan May Be Hard Sell," warned the Minneapolis Star-Tribune. Commentator Kevin Phillips cited the 22 percent figure as evidence that Bush's budget was "contemptuous of public opinion." Reuters artificially lumped together the three non-tax-cut alternatives offered in the Post's question, concluding, "Seventy-seven percent cited other priorities."
Readers should especially beware comparative questions disguised as yes/no questions. On March 8, the front page of the Los Angeles Times ("Public Supports Bush, Is Divided on His Tax Cut Plan") reported that according to a new Times poll, only "a narrow majority of Americans now says it would support Bush's proposed $1.6-trillion tax cut." Readers who examined the Times' separate 4,000-word analysis of the poll were told, "Forty percent disapprove of [Bush's] plan." Only those who insisted on seeing the raw questionnaire discovered that the question actually asked respondents whether they approved of Bush's tax cut "rather than a targeted tax cut for low- and middle-income families."
3. How are the data categorized? Lumping together two categories of responses is an easy way to make them look bigger. IMC told the media that in its poll, "A combined 52% (43% right size, 9% should be bigger) believe the Bush tax cut is either the right size or should be larger." IMC didn't mention, however, that 30 percent of respondents said the tax cut should be smaller and that, therefore, a combined 73 percent believed the tax cut was either the right size or should be smaller. Pollsters also know that respondents, given a spectrum of answers, will gravitate toward the center. If IMC had wanted results favoring the Democratic tax cut proposal, it would have put Bush's plan at the high end of the spectrum, "no tax cut" at the low end, and the Democratic plan in the middle.
Newsweek's recent survey used the same lumping device in reverse. It made the pro-tax-cut constituency look small by lumping together the alternatives in a bipolar question: "In general, would you prefer to see the federal surplus used mainly to cut people's taxes or to pay down the federal debt and make Social Security and Medicare more financially solvent?" Naturally, tax cuts got creamed. How many people would have preferred a combination of tax cuts and debt repayment or tax cuts and shoring up Social Security? We don't know, because those possibilities weren't offered. But the gang-up format of the question escaped the media's notice. The Post reported that 65 percent of Newsweek respondents "favor using projected budget surpluses to reduce the national debt."
4. Are the comparisons fair? A week ago, Gallup proclaimed that in its own survey, "the public was asked to rate each of six different priorities for the Bush administration. As can be seen, 'cutting federal income taxes' comes in dead last." That figure quickly permeated the media. "Out of six public policies tested, cutting federal income taxes ranked last," reported the New York Times. But the question was horribly skewed. The five alternatives that beat out tax cuts were "improving education," "keeping America prosperous," "dealing with the energy problems facing the nation," "keeping the federal budget balanced," and "providing military security for the country." Those aren't policies. They're objectives. Worse, they're objectives nobody opposes. Any disputed policy served up on a list with them was certain to come in last.
According to the Houston Chronicle ("Many Show Skepticism Over Tax Cut"), a recent Reuters/Zogby poll "found that voters ranked cutting taxes fourth in their priorities, behind funding programs for working families, saving Social Security and paying down the national debt." The precise wording of the fourth option offered to respondents was "programs that help working families, such as child care, expanded health care coverage, and education." Naturally, this option won. Cutting taxes trailed debt repayment by a single, statistically insignificant percentage point, but was pronounced the loser.
5. What's the trend? "It has become increasingly clear that Bush still confronts significant public and legislative skepticism" about his tax cut, the Los Angeles Times asserted on Feb. 27, citing the Pew survey. Other news outlets noted that tax cuts came in third out of four options in the poll, with just 19 percent support. Increasingly clear? The media failed to mention that this was double the percentage who had chosen tax cuts on the same question in the same poll three years earlier. Meanwhile, the Post, outlining its own survey findings, trumpeted " new indications that [Bush] will have a difficult time achieving passage of key parts of his agenda. … Only 22 percent favored a tax cut as the top priority." The Dallas Morning News told its readers that the Post survey "found tax relief falling behind increased spending for education and other domestic programs and strengthening Social Security." Neither article acknowledged that the 22 percent figure was up from 14 percent in the same poll five months earlier.
Does this mean you can't believe poll numbers? No. It just means you have to scrutinize them like budget numbers. Accept no paraphrases or characterizations. Demand to see the exact questions. And bring a few questions of your own.