Trailhead: A campaign blog.

Pollsters Protect Magic Algorithms

A quick assessment of pre-election polling versus election results over at RealClearPolitics shows a typical sampling of accuracy and error in RCP's average of major polls. (See chart at end of post.) But social psychologist Jon Krosnick, a professor at Stanford and an expert on polling methodology, points out that, whether the various polls are dead-on or egregiously off-base at the end of the night, we still will not learn anything about how to do it better next time.

The problem, Krosnick said when I caught up with him this evening, is that pollsters refuse to release their methodology after the fact. There are enough variables in the process -- most importantly, how the pollsters defines the slippery concept of a "likely voter" -- that it is very difficult to independently assess which are more robust.

"What it comes down to is that the people who are making money by doing polling don’t want to reveal anything that can be used against them," Krosnick says. "But if they’re reluctant now, why shouldn’t they release the [complete] polls they did five years ago?"

Simply releasing the complete data sets, not just the conclusions, he says, would allow him and his colleagues to reconstruct the methodology and compare it to actual voters in previous elections, enabling them to determine which estimates of likely voters were most accurate. As it stands, he says, "there’s really no potential for us to have a solid scientific basis to determine this."