After musing this morning on different ways of thinking about what a TV ad can do, I’d be remiss if I didn’t note two sources that have spent the summer auditioning new tools to assess the success of a single spot.
Vanderbilt University political scientist John Geer has teamed up with YouGov, a pollster that uses Internet-based samples, and launched the Ad Rating Project, which allows a 600-person panel to judge ads. The project is designed to be a democratized counterweight to punditry. “We no longer have to rely solely on ad-hoc conversations among experts, various fact checkers, and journalists about whether an ad crosses the line,” Geer and YouGov’s Doug Rivers wrote in Politico. “We instead give Americans a chance to weigh in with their thoughts.” As a consequence, it relies on purely qualitative questions. Did viewers like the ad? Was it memorable, interesting, unfair, untruthful or unbelievable? Did it make them feel hopeful, happy, disgusted, angry, and worried?
These are, of course, foremost judgments about aesthetics and propriety. Adam Schaeffer, a co-founder and research director of the Republican consulting firm Evolving Strategies, is trying to measure impact.
His firm is promoting the use of what it brands a “PocketTrial,” basically a lab experiment, in which an online sample is randomly shown ads and then polled on the state of the race and candidate popularity afterwards. Since the viewers were assigned to groups randomly, Schaeffer can attribute differences in how the groups respond to the influence of the ads they saw. “This is the only way you can get some purchase on causality,” he says.
When he recently showed two short videos to his subjects to assess the impact of messaging related to Paul Ryan’s selection—one from Romney’s campaign trumpeting the ticket, the other from Obama’s attacking the Ryan Plan—in each case male viewers moved dramatically in the directions each ad wanted to push them while female ones barely budged. “Women seem more stable in their vote choice,” he says.
Schaeffer is reassuringly timid about venturing an assumption as to why that might be. But he scoffs at the idea that the answer could be uncovered by asking voters to declare whether they find an ad interesting or claim to be disgusted by it. “We just observe—we don’t ask people to judge an ad,” he says. “We don’t care what they think of it.”