Informed Speculation

Well, it was inevitable…every time I click through to a lefty source, whether it’s homegrown rag the Austin Chronicle or nationwide propaganda machine “Democracy Now!”, I’m confronted with that damn Lancet study with its assertions that 700 Iraqis a day have died violent deaths because of the invasion over a period of 3 years. I didn’t believe it then, and I believe it less now, particularly given the widespread media coverage of the horrible toll recently, when there have actually been single days with over 100 casualties, a terribly regular occurence of late. 

Nevertheless, I took considerable heat from my critics for asserting the ‘common sense’ defense, so I feel obligated (nah, that’s not true – I just like doing it) to point out criticisms of the study that are built on a more solid foundation.  One such criticism of recent vintage came from Slate’s Fred Kaplan:

There are two reasons to suspect that the sample was not random, and one of those reasons suggests that the sample was biased in a way that exaggerates the death toll.

First, the Lancet study, like all such studies, estimates not how many people have died, but rather the difference between how many people died in a comparable period before the invasion and how many people have died since the invasion. As the study puts it, 655,000 is roughly the number of deaths “above the number that would be expected in a non-conflict situation.”

In any such study, it’s crucial that the base-line number—deaths before the invasion—is correct. The Lancet study’s base-line number is dubious.

Based on the household surveys, the report estimates that, just before the war, Iraq’s mortality rate was 5.5 per 1,000. (That is, for every 1,000 people, 5.5 die each year.) The results also show that, in the three and a half years since the war began, this rate has shot up to 13.3 per 1,000. So, the “excess deaths” amount to 7.8 (13.3 minus 5.5) per 1,000. They extrapolate from this figure to reach their estimate of 655,000 deaths.

However, according to data from the United Nations, based on surveys taken at the time, Iraq’s preinvasion mortality rate was 10 per 1,000. The difference between 13.3 and 10.0 is only 3.3, less than half of 7.8.

Does that mean that the post-invasion death toll is less than half of 655,000? Not necessarily. You can’t just take the data from one survey and plug them into another survey. Maybe the Hopkins survey understated post-invasion deaths as much as it understated preinvasion deaths—in which case, the net effect is nil. Maybe not. Either way, it should have been clear to the data-crunchers that something was wrong with the numbers for preinvasion and post-invasion deaths, since they were derived from the same survey.

“When you get these large discrepancies between your own results and results that are already well-established, you recrunch your numbers or you send your survey team back into the field to widen your sample,” Beth Osborne Daponte, a demographer at Yale University who has worked on many studies of this sort, told me in a phone interview. “Obviously, they couldn’t do that here. It’s too dangerous. But that doesn’t change the point. You need to triangulate your data”—to make sure they match other data, or, if they don’t, to figure out why. “They didn’t do that.”

(If the Hopkins researchers want to claim that their estimate is more reliable than the United Nations’, they will have to prove the point. It is also noteworthy that, if Iraq’s preinvasion mortality rate really was 5.5 per 1,000, it was lower than that of almost every country in the Middle East, and many countries in Western Europe.)

This flaw—or discrepancy—doesn’t tell you whether 655,000 is too high, too low, or (serendipitously) just right. It just tells you that something about the number is almost certainly off.

However, the second flaw suggests that the number is almost certainly too high.

A joint research team led by physicists Sean Gourley and Neil Johnson of Oxford University and economist Michael Spagat at Royal Holloway University in London noticed the second flaw. In a statement released Thursday (and reported in today’s issue of the journal Science), they charged that the Lancet study is “fundamentally flawed”—and in a way that systematically overstates the death toll.

The Lancet study, in its section on methodology, notes that the teams picked the houses they would survey from a “random selection of main streets,” defined as “major commercial streets and avenues.” (Italics added.) They also chose from a “list of residential streets crossing” those main streets.

The Oxford-Holloway team calls this method “main street bias.” They add:

Main street bias inflates casualty rates since conflict events such as car bombs, drive-by shootings, artillery strikes on insurgent positions, and marketplace explosions gravitate toward the same neighborhood types that the [Lancet] researchers surveyed. …

In short, the closer you are to a main road, the more likely you are to die in violent activity. So if researchers only count people living close to a main road, then it comes as no surprise they will over-count the dead.

Whether or not the Hopkins researchers were aware of this flaw, or its importance, is unclear. An exchange of e-mails with Gilbert Burnham, the study’s chief researcher, raises some disturbing questions about this matter. (Click here for the details.)

Aha, you may say, but the Lancet researchers responded to Kaplan’s criticisms.  Well, yes they did…and a feeble response it was.  Ah, well, those that want to believe it will believe it, and those who don’t, won’t.  But somewhere out there is an objective reality where, yes, here I go again, these numbers just don’t make sense…