Download the MP3 audio version of this story here, or sign up for The Explainer’s free daily podcast on iTunes.
A study published in the Lancet this week estimates that 654,965 Iraqis have died as a consequence of war since 2003. President Bush was quick to dismiss the numbers: “I don’t consider it a credible report,” he said on Wednesday. The next day a spokesman for Tony Blair said the figure wasn’t “anywhere near accurate,” and had been extrapolated from a small and unrepresentative sample. So, how good was the sample?
It’s hard to say for sure. The researchers—led by Gilbert Burnham of Johns Hopkins University—gathered data on more than 12,000 people in clusters of houses around Iraq, and tried to figure out how many people had died both before and after March of 2003. By comparing the pre- and post-invasion mortality rates, they figured out how many deaths could be attributed to the war, and then extrapolated from their sample to the country’s entire population.
It’s the same basic method used for political polls in America, which estimate the attitudes of millions of people by surveying 1,000 adults. The trick in a study like this is to make sure that you’re looking at a random sample. If each of the Iraqi interviewees was picked at random, the researchers could make a very precise guess about the total mortality rate across the country. That would show up as a very small margin of error—and a narrow confidence interval around the estimate. The size of the confidence interval reflects their degree of certainty about their guess.
The Hopkins study gives a 95-percent confidence interval that ranges from about 400,000 to almost 950,000. (Click here for a primer on how to interpret these numbers.) Why is there so much uncertainty? The 12,801 Iraqis covered by the study weren’t selected entirely at random. They were drawn from 1,849 households, which were themselves sampled in groups of 40 —with researchers visiting 40 neighboring houses in each of several dozen locations around the country.
When the data points are clustered like this, the statistics become less precise. That’s because you’d expect death rates to be similar for two people who happen to live next door to each other. If two of your subjects live near each other in a high-risk neighborhood in West Baghdad, the information they’re giving you is somewhat redundant. It’s just as if you had a smaller sample to begin with.
The clustering of data points adds some fuzziness to the results of a survey, but it doesn’t have to produce a bias in one direction or another. If you correct for the clustering, all it does is widen the confidence interval. There’s nothing unusual or unsound about this technique—international health researchers often resort to cluster samples when more-extensive canvassing proves too dangerous.
The results of a study like this could still be skewed if the clusters weren’t carefully (that is to say, randomly) chosen. For example, you’d see an inflated death toll if the Hopkins researchers had unconsciously chosen more clusters from unsafe areas than from safe ones. The paper also points out that cluster selections were made on the basis of population data from several years ago, and that any significant migrations since then could have introduced some bias. (Click here to read about some other factors that might be relevant.)
Got a question about today’s news? Ask the Explainer.
Explainer thanks Rick Brennan of the International Rescue Committee, Gilbert Burnham and Scott Zeger of Johns Hopkins University, and Gary King of Harvard University.
