Looks like they’re still fairly uncertain about these numbers – kind of interesting when the numbers are tossed around as “facts” in news stories.
From today’s WSJ:
The Numbers Guy
by Carl Bialik
Counting the Civilian Dead in Iraq
August 5, 2005
Thousands of Iraqi civilians have died as a result of violence since the war began in 2003. But it’s not known exactly how many died, where and at whose hands.
There have been several efforts to count the war’s toll on civilians, yielding reports ranging from 24,000 to 128,000 from last fall through last month. Compounding the complexity, all of these numbers were collected differently and count different things, so they aren’t directly comparable. For example, the widely cited number last month of about 25,000 counts only violent deaths that have been reported to the media. Meanwhile, a study conducted last fall that found 100,000 deaths arrived at that figure by calculating “excess” deaths – all deaths, including those from illness and accidents, were included, but deaths from a comparable prewar period were subtracted out.
The uncertain and inconsistent numbers help explain why the civilian death toll – caused by criminals, terrorists, insurgents and soldiers from all sides – hasn’t been given much attention in major U.S. media, even as many newspapers report every death of U.S. soldiers in Iraq and, last December, headlined incomplete tsunami death-toll numbers for weeks.
Accurate death counts of civilians are often a casualty of war. Those best positioned to collect the numbers – governments and military forces – often have an interest in manipulating civilian casualty counts, and any numbers they report would be questioned because of bias. As Patrick Ball, who has studied civilian deaths in Peru, Guatemala and Kosovo, told me in an email, “It turns out that the people who commit mass political violence are not generally in favor of careful, post-hoc documentation and statistical analysis.” Meanwhile, wartime conditions dissuade most independent researchers, and reporting systems at hospitals and morgues are often disrupted. Separating combatants from noncombatants is also difficult.
Yet more accurate numbers would be valuable, both politically and practically. “They could bring some accuracy and some temperance to the far-flung allegations, both from the left and from the right,” about the war’s toll, Sarah Sewall, program director at Harvard University’s Carr Center for Human Rights Policy, told me. Knowing how many Iraqis are dying can help people around the world decide if the war has been worthwhile for Iraq and is being conducted in the best way. More importantly for Iraqis, identifying the civilian dead “could direct assistance and compensation efforts,” Ms. Sewall says. And learning why and how civilians have died could help the U.S.-led forces in Iraq, and planners of future wars, minimize the death of innocents.
Monitoring Press Reports
Last month, a mostly volunteer, United Kingdom-based group called Iraq Body Count reported that 24,865 civilians had died in violence since the war began ( Iraq Body Count Press Release 12 (19 Jul 2005) :: Iraq Body Count ), with 37% of those killed by U.S.-led forces, 9% by insurgents and other anti-occupation forces, and 36% by criminals. The group gathers its numbers from mortuaries and media reports. Media reports could overstate death tolls, as Iraqi Body Count’s critics point out. However, the group only counts incidents that have been reported by at least two separate news organizations, and waits for definitive accounts where available. Also, overblown media death counts cut both ways in IBC’s method, because the group subtracts combatant deaths from its totals. Furthermore, to its credit, IBC is transparent, publishing all its data in an online database ( Iraq Body Count ).
IBC relies on “passive reporting” – it is up to the media and mortuaries to detect deaths, and they inevitably miss some. Muslims often bury their dead before hospitals and mortuaries can learn about the deaths. And about half of the recorded deaths have come in Baghdad, in part a result of the media’s disproportionate representation in the relatively more secure capital city. “It must be incomplete,” John Sloboda, co-founder of IBC, told me, though he said his “gut feeling” is that the group’s estimate represents more than half of all violent deaths.
A more complete accounting of deaths, in a country such as Iraq that doesn’t have available complete official counts, requires a technique akin to political polling: Find a representative sample of Iraqis and ask them to report deaths in their immediate families, then extrapolate to the entire county. One drawback to this method is that a household that gets wiped out entirely can’t report its own deaths. Nonetheless, the method has been used in prior conflicts, including in the Congo. “Epidemiological studies really work in places where getting much data is nearly impossible,” Dr. Ball says.
One such effort, funded by the United Nations Development Programme and carried out by the Norwegian research foundation FAFO and the Iraqi statistical office, interviewed almost 22,000 households throughout Iraq. ( http://www.iq.undp.org/ILCS/PDF/Analytical%20Report%20-%20English.pdf ) The report, released in May, concluded that between 18,000 and 29,000 Iraqi civilians had died because of violence, from the start of the war through May of last year.
But the death question was merely one of dozens, with the average interview lasting 83 minutes. The primary purpose of the survey was to assess living conditions and infrastructure.
Lead researcher Jon Pederson, FAFO’s deputy managing director, also told me that he had reservations about the Iraqi statistical workers who carried out the research – not their impartiality, but their techniques. Statistical workers who have worked under dictatorships “tend to develop a fairly loose relation to data because they know that things will be changed by the government, and that leads to a sloppiness in field work,” Mr. Pederson says, though he adds that the statistical chief, installed after the removal of Saddam Hussein, was committed to accuracy.
Later last year, a team of researchers from Johns Hopkins University, Columbia University and Baghdad’s Al-Mustansiriya University surveyed Iraqis only about mortality. Their goal was to count “excess deaths,” comparing the death rate before the war with the death rate afterwards. All deaths were counted – from violence, illness and accidents. Like in the UNDP study, researchers selected a random sample of Iraqi households, but the sample size was much smaller – about 1,000 households. And security was a big obstacle. While researchers collected some reports from Fallujah, for instances, they ended up discarding that data because fear of violence caused them to collect data differently than in other places, creating a greater potential for faulty numbers.
A Wide Range
Researchers concluded that about 100,000 more Iraqis had died outside Fallujah since the invasion than would have died had the prewar death rate continued. ( Mortality before and after the 2003 invasion of Iraq: cluster sample survey - The Lancet ) Yet the study, published in the British medical journal Lancet, was roundly criticized for discarding the Fallujah data from calculations. Others questioned the study for extrapolating from only 89 death reports outside Fallujah, including reports of 21 violent deaths.
The biggest concern with the Lancet study may be its enormous error range: The study said the number of dead from the war (again, excluding Fallujah) could be as low as 8,000, or as high as 194,000.
Les Roberts, epidemiologist at the Johns Hopkins School of Public Health and lead author of the Lancet paper, said that more than 80% of the time, the first two reported deaths in each neighborhood were confirmed by death certificate, and neighbors often independently corroborated the reports.
The Lancet study was dismissed by many critics, especially supporters of the war, because of two quotes that appeared in early press coverage of the report. Dr. Roberts told the Associated Press ( http://abcnews.go.com/International/wireStory?id=206232 ) that he wanted the study to be published before the election (it was published online by the British medical journal Lancet in late October, just a month after the research was completed), so that “both candidates would be forced to pledge to protect civilian lives in Iraq.” He also said he was opposed to the war. Meanwhile, Marc E. Garlasco, senior military analyst for Human Rights Watch, told the Washington Post ( http://www.washingtonpost.com/wp-dyn/articles/A7967-2004Oct28.html ), “These numbers seem to be inflated.” Many commentators used that quote to argue that even Human Rights Watch, a group that has criticized U.S. forces’ treatment of civilians in the war, didn’t believe the study.
But neither quote provided just cause to dismiss the study. Dr. Roberts and co-author Richard Garfield, a professor of nursing at Columbia University who has extensively studied mass killings, say they had never discussed their views on the war before the AP interview appeared. Both also say that when the five researchers did discuss their views, they found that they were split on the war’s merits.
I’ve written before about the importance of researcher bias, but in this case there’s no indication that the research methods were chosen in a way that would intentionally inflate the numbers. Other researchers have told me and other reporters that the methods were sound; Dr. Roberts has experience measuring death tolls, including in the Congo, and his numbers have been widely accepted in less politicized conflicts. Furthermore, there’s a difference between researchers letting their opinions affect their results and letting their results affect their opinions.
As for Mr. Garlasco’s quote, he has since said that he hadn’t yet read the study when he was interviewed. He told me last week, “I think the Lancet study is very valuable in some ways,” and added, “I’m not a statistician. I don’t really understand statistics. I try to stay away from numbers as much as possible.”
Better Numbers
Is there hope for better numbers? Some have called on the Pentagon to report civilian deaths from violence, pointing out that the U.S. military did so in some past conflicts – for example, reporting 200 Panamanian civilians killed in the 1989 invasion. However, in larger conflicts since then – including the first Gulf War – the military hasn’t reported civilian deaths. U.S. officers in Iraq have told reporters that they do file incident reports about civilian deaths, and the military also considers wrongful-death claims from Iraqis, in some cases approving compensation. But the military hasn’t shared its numbers. The Pentagon’s Coalition Press Information Center didn’t respond to my request for comment, made last Friday. “Without a doubt, the U.S. military is in the best position to provide this kind of information,” Human Rights Watch’s Mr. Garlasco says.
Several researchers told me that the best bet for a complete account would be an effort funded by an international body like the United Nations or the World Health Organization, modeled after the Lancet study but with a far greater sample size and better accounting for high-violence areas like Fallujah. (Spokeswomen for the WHO and the UNDP said they didn’t know of plans for their organizations to fund mortality studies.) Dr. Roberts says he had hoped and expected more studies to follow his own group’s, with more money. “What I did was on a shoestring and under the radar,” he says. He recommends a larger sample, and also dividing the country into three areas – the Kurdish north, where his study found that mortality had declined since the war; the cities that have seen the most violence, including Fallujah; and the rest of the country – and measuring deaths in each separately, then combining the results, which would decrease statistical uncertainty. “We absolutely need a record of everyone who died,” Dr. Roberts says.
