Taking another look at the Lancet's Iraqstudy.
Let us take another look at the Lancet study. This is the report, issued by a team from Johns Hopkins University and published in the current issue of British medical journal the Lancet, estimating that 655,000 Iraqis have died as a consequence of the U.S.-led invasion. It's a shocking number. Is it true?
Initially, I decided to stay out of this controversy. I'd written the first critique of an earlier Lancet/Hopkins study, which estimated that 100,000 Iraqis had died in just the first year of the war. The study's sample was too small, the data-gathering too slipshod, the range of uncertainty so wide as to render the estimate useless.
The new study looked better: a larger sample, more fastidious attention to data-gathering procedures, a narrower range of uncertainty. The number—655,000 deaths—seemed improbably high (that's an average of 20,000 deaths a month since the war began), but so have a lot of savage statistics that turned out to be true; and there are many areas of Iraq these days where reporters and human rights groups dare not roam.
However, the more I read the study and the more I talked with statisticians, the flimsier this number appeared. The study might be as good an effort as anyone can manage in wartime. Certainly, the Iraqis who went door to door conducting the surveys are amazingly brave souls. But the study has two major flaws—the upshot of which is that it's impossible to infer anything meaningful from it, except that a lot of Iraqis have died and the number is getting higher.
This point should be emphasized. Let's say that the study is way off, off by a factor of 10 or five—in other words, that the right number isn't 655,000 but something between 65,500 and 131,000. That is still a ghastly number—a number that, apart from all other considerations, renders this war a monumental mistake. Here's the key question: Had it been known ahead of time that invading Iraq would result in the deaths of 100,000 Iraqis (or 50,000, or pick your own threshold number), would the president have made—would Congress have voted to authorize, would any editorial writer or public figure have endorsed—a decision to go to war?
Here lies the danger of studies that overstate a war's death toll. The war's supporters and apologists latch on to the inevitable debunkings and proclaim that really "only 100,000" or "only 200,000" people have died. It's obscene—it sullies and coarsens the political culture—to place the word "only" in front of such numbers.
So, let's look at this study's numbers and why they're almost certainly overstated.
The researchers reached this conclusion through a common technique known as "cluster sampling." They randomly selected 47 neighborhoods in 18 of Iraq's regions. Within those neighborhoods, they visited a total of 1,849 households, comprising 12,801 residents, and asked how many of their members had died before the invasion and since the invasion. The researchers then extrapolated from this sample to the entire Iraqi population of 27 million people—from which they concluded that since the war there have been about 655,000 "excess deaths," of which 601,000 were caused by violence.
This methodology is entirely proper if the sample was truly representative of the entire population—i.e., as long as those households were really randomly selected. If they were not randomly selected—if some bias crept into the sampling, even unintentionally—then it is improper, and wildly misleading, to extrapolate the findings to the population as a whole.
There are two reasons to suspect that the sample was not random, and one of those reasons suggests that the sample was biased in a way that exaggerates the death toll.
Fred Kaplan is Slate's "War Stories" columnist and author of the book, The Insurgents: David Petraeus and the Plot to Change the American Way of War. He can be reached at email@example.com. Follow him on Twitter.