The authors of a peer-reviewed study, conducted by a survey team from Johns Hopkins University, claim that about 100,000 Iraqi civilians have died as a result of the war. Yet a close look at the actual study, published online today by the British medical journal the Lancet, reveals that this number is so loose as to be meaningless.
The report's authors derive this figure by estimating how many Iraqis died in a 14-month period before the U.S. invasion, conducting surveys on how many died in a similar period after the invasion began (more on those surveys later), and subtracting the difference. That difference—the number of "extra" deaths in the post-invasion period—signifies the war's toll. That number is 98,000. But read the passage that cites the calculation more fully:
We estimate there were 98,000 extra deaths (95% CI 8000-194 000) during the post-war period.
Readers who are accustomed to perusing statistical documents know what the set of numbers in the parentheses means. For the other 99.9 percent of you, I'll spell it out in plain English—which, disturbingly, the study never does. It means that the authors are 95 percent confident that the war-caused deaths totaled some number between 8,000 and 194,000. (The number cited in plain language—98,000—is roughly at the halfway point in this absurdly vast range.)
This isn't an estimate. It's a dart board.
Imagine reading a poll reporting that George W. Bush will win somewhere between 4 percent and 96 percent of the votes in this Tuesday's election. You would say that this is a useless poll and that something must have gone terribly wrong with the sampling. The same is true of the Lancet article: It's a useless study; something went terribly wrong with the sampling.
The problem is, ultimately, not with the scholars who conducted the study; they did the best they could under the circumstances. The problem is the circumstances. It's hard to conduct reliable, random surveys—and to extrapolate meaningful data from the results of those surveys—in the chaotic, restrictive environment of war.
However, these scholars are responsible for the hype surrounding the study. Gilbert Burnham, one of the co-authors, told the International Herald Tribune (for a story reprinted in today's New York Times), "We're quite sure that the estimate of 100,000 is a conservative estimate." Yet the text of the study reveals this is simply untrue. Burnham should have said, "We're not quite sure what our estimate means. Assuming our model is accurate, the actual death toll might be 100,000, or it might be somewhere between 92,000 lower and 94,000 higher than that number."
Not a meaty headline, but truer to the findings of his own study.
Here's how the Johns Hopkins team—which, for the record, was led by Dr. Les Roberts of the university's Bloomberg School of Public Health—went about its work. They randomly selected 33 neighborhoods across Iraq—equal-sized population "clusters"—and, this past September, set out to interview 30 households in each. They asked how many people in each household died, of what causes, during the 14 months before the U.S. invasion—and how many died, of what, in the 17 months since the war began. They then took the results of their random sample and extrapolated them to the entire country, assuming that their 33 clusters were perfectly representative of all Iraq.