Number Crunching

Military analysis.
Oct. 20 2006 6:34 PM

Number Crunching

Taking another look at the Lancet's Iraqstudy.

Let us take another look at the Lancet study. This is the report, issued by a team from Johns Hopkins University and published in the current issue of British medical journal the Lancet, estimating that 655,000 Iraqis have died as a consequence of the U.S.-led invasion. It's a shocking number. Is it true?

Initially, I decided to stay out of this controversy. I'd written the first critique of an earlier Lancet/Hopkins study, which estimated that 100,000 Iraqis had died in just the first year of the war. The study's sample was too small, the data-gathering too slipshod, the range of uncertainty so wide as to render the estimate useless.

The new study looked better: a larger sample, more fastidious attention to data-gathering procedures, a narrower range of uncertainty. The number—655,000 deaths—seemed improbably high (that's an average of 20,000 deaths a month since the war began), but so have a lot of savage statistics that turned out to be true; and there are many areas of Iraq these days where reporters and human rights groups dare not roam.

However, the more I read the study and the more I talked with statisticians, the flimsier this number appeared. The study might be as good an effort as anyone can manage in wartime. Certainly, the Iraqis who went door to door conducting the surveys are amazingly brave souls. But the study has two major flaws—the upshot of which is that it's impossible to infer anything meaningful from it, except that a lot of Iraqis have died and the number is getting higher.

This point should be emphasized. Let's say that the study is way off, off by a factor of 10 or five—in other words, that the right number isn't 655,000 but something between 65,500 and 131,000. That is still a ghastly number—a number that, apart from all other considerations, renders this war a monumental mistake. Here's the key question: Had it been known ahead of time that invading Iraq would result in the deaths of 100,000 Iraqis (or 50,000, or pick your own threshold number), would the president have made—would Congress have voted to authorize, would any editorial writer or public figure have endorsed—a decision to go to war?

Here lies the danger of studies that overstate a war's death toll. The war's supporters and apologists latch on to the inevitable debunkings and proclaim that really "only 100,000" or "only 200,000" people have died. It's obscene—it sullies and coarsens the political culture—to place the word "only" in front of such numbers.

So, let's look at this study's numbers and why they're almost certainly overstated.

The researchers reached this conclusion through a common technique known as "cluster sampling." They randomly selected 47 neighborhoods in 18 of Iraq's regions. Within those neighborhoods, they visited a total of 1,849 households, comprising 12,801 residents, and asked how many of their members had died before the invasion and since the invasion. The researchers then extrapolated from this sample to the entire Iraqi population of 27 million people—from which they concluded that since the war there have been about 655,000 "excess deaths," of which 601,000 were caused by violence.

This methodology is entirely proper if the sample was truly representative of the entire population—i.e., as long as those households were really randomly selected. If they were not randomly selected—if some bias crept into the sampling, even unintentionally—then it is improper, and wildly misleading, to extrapolate the findings to the population as a whole.

There are two reasons to suspect that the sample was not random, and one of those reasons suggests that the sample was biased in a way that exaggerates the death toll.

First, the Lancet study, like all such studies, estimates not how many people have died, but rather the difference between how many people died in a comparable period before the invasion and how many people have died since the invasion. As the study puts it, 655,000 is roughly the number of deaths "above the number that would be expected in a non-conflict situation."

In any such study, it's crucial that the base-line number—deaths before the invasion—is correct. The Lancet study's base-line number is dubious.

Based on the household surveys, the report estimates that, just before the war, Iraq's mortality rate was 5.5 per 1,000. (That is, for every 1,000 people, 5.5 die each year.) The results also show that, in the three and a half years since the war began, this rate has shot up to 13.3 per 1,000. So, the "excess deaths" amount to 7.8 (13.3 minus 5.5) per 1,000. They extrapolate from this figure to reach their estimate of 655,000 deaths.

However, according to data from the United Nations, based on surveys taken at the time, Iraq's preinvasion mortality rate was 10 per 1,000. The difference between 13.3 and 10.0 is only 3.3, less than half of 7.8.

Does that mean that the post-invasion death toll is less than half of 655,000? Not necessarily. You can't just take the data from one survey and plug them into another survey. Maybe the Hopkins survey understated post-invasion deaths as much as it understated preinvasion deaths—in which case, the net effect is nil. Maybe not. Either way, it should have been clear to the data-crunchers that something was wrong with the numbers for preinvasion and post-invasion deaths, since they were derived from the same survey.

"When you get these large discrepancies between your own results and results that are already well-established, you recrunch your numbers or you send your survey team back into the field to widen your sample," Beth Osborne Daponte, a demographer at Yale University who has worked on many studies of this sort, told me in a phone interview. "Obviously, they couldn't do that here. It's too dangerous. But that doesn't change the point. You need to triangulate your data"—to make sure they match other data, or, if they don't, to figure out why. "They didn't do that."

(If the Hopkins researchers want to claim that their estimate is more reliable than the United Nations', they will have to prove the point. It is also noteworthy that, if Iraq's preinvasion mortality rate really was 5.5 per 1,000, it was lower than that of almost every country in the Middle East, and many countries in Western Europe.)

This flaw—or discrepancy—doesn't tell you whether 655,000 is too high, too low, or (serendipitously) just right. It just tells you that something about the number is almost certainly off.

However, the second flaw suggests that the number is almost certainly too high.

A joint research team led by physicists Sean Gourley and Neil Johnson of Oxford University and economist Michael Spagat at Royal Holloway University in London noticed the second flaw. In a statement released Thursday (and reported in today's issue of the journal Science), they charged that the Lancet study is "fundamentally flawed"—and in a way that systematically overstates the death toll.

The Lancet study, in its section on methodology, notes that the teams picked the houses they would survey from a "random selection of main streets," defined as "major commercial streets and avenues." (Italics added.) They also chose from a "list of residential streets crossing" those main streets.

The Oxford-Holloway team calls this method "main street bias." They add:

Main street bias inflates casualty rates since conflict events such as car bombs, drive-by shootings, artillery strikes on insurgent positions, and marketplace explosions gravitate toward the same neighborhood types that the [Lancet] researchers surveyed. …

In short, the closer you are to a main road, the more likely you are to die in violent activity. So if researchers only count people living close to a main road, then it comes as no surprise they will over-count the dead.

Whether or not the Hopkins researchers were aware of this flaw, or its importance, is unclear. An exchange of e-mails with Gilbert Burnham, the study's chief researcher, raises some disturbing questions about this matter. (Click here for the details.)

It's understandable why the surveyors limited their work to the main roads; they were in strange and dangerous places. But that doesn't negate the Oxford-Holloway team's point. By this measure alone, the Lancet study is not a random survey. In statistically proper random surveys, each household has the same probability of being chosen. Yet in the Lancet survey, if a household wasn't on or near a main road, it had zero chance of being chosen. And "cluster samples" cannot be seen as representative of the entire population unless they are chosen randomly.

The Iraq war is a catastrophe in political, military, and—not least—human terms. How much so may be unfathomable as long as the streets of Iraq are still dangerous. In any event, it's a question that the Lancet study doesn't really answer.

TODAY IN SLATE

Politics

The Democrats’ War at Home

How can the president’s party defend itself from the president’s foreign policy blunders?

An Iranian Woman Was Sentenced to Death for Killing Her Alleged Rapist. Can Activists Save Her?

Piper Kerman on Why She Dressed Like a Hitchcock Heroine for Her Prison Sentencing

Windows 8 Was So Bad That Microsoft Will Skip Straight to Windows 10

Homeland Is Good Again! For Now.

Politics

Cringing. Ducking. Mumbling.

How GOP candidates react whenever someone brings up reproductive rights or gay marriage.

Music

How Even an Old Hipster Can Age Gracefully

On their new albums, Leonard Cohen, Robert Plant, and Loudon Wainwright III show three ways.

The U.S. Has a New Problem in Syria: The Moderate Rebels Feel Like We’ve Betrayed Them

We Need to Talk: A Terrible Name for a Good Sports Show by and About Women

Trending News Channel
Oct. 1 2014 1:25 PM Japanese Cheerleader Robots Balance and Roll Around on Balls
  News & Politics
The World
Oct. 1 2014 12:20 PM Don’t Expect Hong Kong’s Protests to Spread to the Mainland
  Business
Moneybox
Oct. 1 2014 2:16 PM Wall Street Tackles Chat Services, Shies Away From Diversity Issues 
  Life
The Eye
Oct. 1 2014 1:04 PM An Architectural Crusade Against the Tyranny of Straight Lines
  Double X
The XX Factor
Oct. 1 2014 2:08 PM We Need to Talk: Terrible Name, Good Show
  Slate Plus
Behind the Scenes
Oct. 1 2014 3:24 PM Revelry (and Business) at Mohonk Photos and highlights from Slate’s annual retreat.
  Arts
Brow Beat
Oct. 1 2014 3:02 PM The Best Show of the Summer Is Getting a Second Season
  Technology
Future Tense
Oct. 1 2014 3:01 PM Netizen Report: Hong Kong Protests Trigger Surveillance and Social Media Censorship
  Health & Science
Science
Oct. 1 2014 2:36 PM Climate Science Is Settled Enough The Wall Street Journal’s fresh face of climate inaction.
  Sports
Sports Nut
Sept. 30 2014 5:54 PM Goodbye, Tough Guy It’s time for Michigan to fire its toughness-obsessed coach, Brady Hoke.