Liberal Stats on Iraqi Civilian Deaths is Found to be Way to High
By Neil Munro and Carl M. Cannon, National Journal
© National Journal Group Inc.
Friday, Jan. 4, 2008 Three weeks before the 2006 midterm elections gave Democrats control of Congress, a shocking study reported on the number of Iraqis who had died in the ongoing war. It bolstered criticism of President Bush and heightened the waves of dread — here and around the world — about the
Published by The Lancet, a venerable British medical journal, the study [PDF] used previously accepted methods for calculating death rates to estimate the number of “excess” Iraqi deaths after the 2003 invasion at 426,369 to 793,663; the study said the most likely figure was near the middle of that range: 654,965. Almost 92 percent of the dead, the study asserted, were killed by bullets, bombs, or
In December 2005, Bush had used a figure of 30,000 civilian deaths in
Queried in the Rose Garden on October 11, the day the Lancet article came out, Bush dismissed it. “I don’t consider it a credible report,” he replied. The Pentagon and top British government officials also rejected the study’s findings.
Such skepticism would not prove to be the rule.
CBS News called the report a “new and stunning measure of the havoc the American invasion unleashed in
Editorials in many major newspapers cited the Lancet article as further evidence that the invasion of
Within a few weeks a backlash rose, although the contrarian view of the study generated far less press attention than the Lancet article. In the ensuing year, numerous skeptics have identified various weaknesses with the study’s methodology and conclusions. Political blogs and academic journals have registered and responded to the objections in a debate that has been simultaneously arcane and predictable. The arguments are arcane because that is the nature of statistical analysis. They are predictable because that is the nature of today’s polarized political discourse, with liberals defending the Lancet study and conservatives contesting it.
How to explain the enormous discrepancy between The Lancet’s estimation of Iraqi war deaths and those from studies that used other methodologies? For starters, the authors of the Lancet study followed a model that ensured that even minor components of the data, when extrapolated over the whole population, would yield huge differences in the death toll. Skeptical commentators have highlighted questionable assumptions, implausible data, and ideological leanings among the authors, Gilbert Burnham, Riyadh Lafta, and Les Roberts.
Some critics go so far as to suggest that the field research on which the study is based may have been performed improperly — or not at all. The key person involved in collecting the data — Lafta, the researcher who assembled the survey teams, deployed them throughout
Some of these questions could be resolved if other researchers had access to the surveyors’ original field reports and response forms. The authors have released files of collated survey results but not the original survey reports, citing security concerns and the fact that some information was not recorded or preserved in the first place. This was a legitimate problem, and it underscored the difficulty of conducting research in a war zone.
Each death recorded by the
Over the past several months, National Journal has examined the 2006 Lancet article, and another [PDF] that some of the same authors published in 2004; probed the problems of estimating wartime mortality rates; and interviewed the authors and their critics. NJ has identified potential problems with the research that fall under three broad headings: 1) possible flaws in the design and execution of the study; 2) a lack of transparency in the data, which has raised suspicions of fraud; and 3) political preferences held by the authors and the funders, which include George Soros’s Open Society Institute. Origins Of The Survey Since the beginning of the war, the media have meticulously tracked and documented the number of American soldiers killed in Iraq — which reached 3,904 on January 1 — particularly as the total approached and then surpassed (in December 2006) the 2,973 people killed in the 9/11 terrorist attacks. But determining the number of Iraqis who have died is much more difficult, as is determining how many of the dead were insurgents and how many were innocent civilians. With
The Lancet study was based on techniques developed by public health experts to determine rates of illness and death from epidemics and famines in large populations. This “cluster” sampling is a relatively new methodology that attempts to replicate the logic of public opinion polling in
Following this method, questioners undertake a house-to-house survey in certain areas and then extrapolate the results from that statistical sample to the entire national population. According to this study’s design, teams of Iraqi questioners would visit approximately 47 randomly chosen clusters of homes throughout the country and ask a series of census-style questions at 40 contiguous households in each cluster: How many people live in your household? How many lived here on January 1, 2002? In that time, how many were born — and how many died?
In 2004, several of the same authors had done a preliminary
The Authors The origins of the Lancet studies can be traced to 1993, when two officials from the U.S. Centers for Disease Control and Prevention traveled to Bosnia-Herzegovina to view the devastation caused by the Balkan war. Only nine years after
In pursuit of an accurate picture, the
“I think that’s when I fully understood the need to step beyond peer-review journals and statistical analyses if you are going to do effective public health work in times of war,” Roberts explained in a recent interview with a Belgian-based publication. This determination to become an advocate would lead him to Rwanda and the Congo, where in 2001 he was involved in studies that produced jaw-dropping estimates of more than 3 million dead in that nation’s civil war. Roberts also went back to the Balkans — this time to Kosovo — and ultimately, when war came to
By then, Roberts was a researcher at the
A car bomb attack in
Lafta had been a child-health official in Saddam Hussein’s ministry of health when the ministry was trying to end the international sanctions against Iraq by asserting that many Iraqis were dying from hunger, disease, or cancer caused by spent U.S. depleted-uranium shells remaining from the 1991 Persian Gulf War. In 2000, Lafta authored at least two brief articles contending that U.N. sanctions had caused many deaths by starvation among Iraqi children. In one article, he identified malnutrition as the main contributor to 53 percent of deaths among hospitalized children younger than 2, during a 1997 survey carried out at Saddam Central Teaching Hospital. The article cited no health data from before the sanctions, yet it asserted, “We can conclude from results that the most important and widespread underlying cause of the deterioration of child-health standards in Iraq is the long-term impact of the nonhumanized economic sanction imposed through United Nations resolutions.” The article was published in 2000 by the Iraqi Journal of Community Medicine. Roberts told National Journal he had not read Lafta’s articles, and Burnham said he did not have a copy of the articles.
Lafta is now at
Lafta and his surveyors often worked under brutal political pressure. In January 2007, a Sunni suicide bomber killed more than 70 students at the university, partly because it is perceived as being under the control of Moktada al-Sadr, the Shiite religious leader whose Mahdi Army militia crippled Sunni insurgent groups in
Dramatic Findings In his first study of Iraqi war deaths, in September 2004, Lafta sent six Iraqi questioners to 33 clusters of homes throughout the country to ask how many people in each household had died since January 1, 2002. The researchers reported that 808 of the 998 identified households participated in the survey, and then extrapolated the number of deaths reported to the entire population of 24.4 million Iraqis. “Making conservative assumptions, we think that about 100,000 excess deaths or more have happened since the 2003 invasion of Iraq,” concluded the authors — Roberts, Lafta, Garfield, Jamal Khudhairi, and Burnham. That was when the war was just 19 months old.
“Violence accounted for most of the excess deaths, and air strikes from coalition forces accounted for most violent deaths,” the report said. According to subsequent explanations by the authors, the total included 57,600 dead from violence, 24,000 dead from wartime accidents, and 13,600 dead from disease. The accidental deaths included 15,000 Iraqis killed by
Little is known about Lafta’s decision-making in amassing the data for the Lancet surveys. Roberts provided some information, however, about Lafta’s 2004 survey of casualties in Falluja. At the time, al-Sadr was publicly supporting the anti-American Sunni radicals who controlled the city. In September, Roberts said, he pleaded with “his Muslim friend Lafta not to go” into Falluja, according to an interview with a magazine published by Johns Hopkins. Roberts told the interviewer that Lafta replied, “God has picked these clusters. If God wants me, he will take me. I must go.” Roberts also said of Lafta, “I know no one [who] perceives themselves so humbly to be a tool of God’s destiny…. He sees his science as synonymous with service to God.”
In Falluja, Lafta recorded 52 deaths in 29 households, which amounted to 71 percent of the violent deaths recorded by the first Lancet survey. If representative, Lafta’s sample translated into 50,000 to 70,000 dead in Falluja by September 2004 — two months before the start of the second major American military operation to restore order. Falluja’s prewar population was estimated to be 250,000, although
The 2006 study, known as Lancet II, was somewhat larger, involving 47 clusters and using similar survey techniques. In all, 302 violent deaths reported in those 1,849 households became the basis for estimating that 601,000 Iraqis had died violently from the start of the war through June 2006.
Even though the second study was even further out of line with other sources’ estimates than the first, it got tremendous attention — probably because its findings fit an emerging narrative:
Democrats who had opposed Bush’s
The Lancet II article was also publicized widely overseas, especially in the
Muslim commentators in the
In the
The John Hopkins researcher, Les Roberts, began the studies by smuggling himself into
The study had such a significant impact partly because of where it appeared. The Lancet, founded in 1823, is one of the world’s most-cited medical journals, credited with publishing articles that established the principles of antiseptics in 1867 and documented the dangers of thalidomide in 1961. Although few mainstream journalists ever plow through the journal’s articles, news outlets typically refer to it as “the respected Lancet.” In recent years, however, the journal’s reputation has suffered from charges of politicization and a few prominent instances of scientific fraud.
Also driving the press attention was the study’s association with Johns Hopkins University, whose School of Public Health was the first and is now the largest such institution in the world. Faculty members participated in the study, and the school’s review board conducted an ethical review of the research plan. The Arab American’s Siblani said that the university connection was one reason he put the study on the front page of his newspaper.
Potential Problems Both Lancet studies of Iraqi war deaths rest on the data provided by Lafta, who operated with little American supervision and has rarely appeared in public or been interviewed about his role. In May, Lafta and Roberts presented their study to an off-the-record meeting of experts in
When asked questions about the reliability of their Iraqi partner, the studies’ American authors defend Lafta as a nice guy and a good researcher.
“I’ve known him for years,”
John Tirman, a political scientist at the Massachusetts Institute of Technology, described Lafta as “a medical doctor, a professor of medicine. Those factors were a sufficient level of credibility. I never asked [Lafta] about his political views.” Tirman commissioned the Lancet II survey with $46,000 from George Soros’s Open Society Institute and additional support from other funders.
Lancet Editor Richard Horton shares this fundamental faith in scientists. He told NJ that scientists, including Lafta, can be trusted because “science is a global culture that operates by a set of norms and standards that are truly international, that do not vary by culture or religion. That’s one of the beautiful aspects of science — it unifies cultures, not divides them.”
Still, the authors have declined to provide the surveyors’ reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent — and repeatable — to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles?
“The authors refuse to provide anyone with the underlying data,” said David Kane, a statistician and a fellow at the Institute for Quantitative Social Statistics at
Fritz Scheuren, vice president for statistics at the National Opinion Research Center and a past president of the American Statistical Association, said, “They failed to do any of the [routine] things to prevent fabrication.” The weakest part of the Lancet surveys is their reliance on an unsupervised Iraqi survey team, contended Scheuren, who has recently trained survey workers in
When the study came out in October 2006, President Bush said it wasn’t credible.
The research is “a field study in unstable conditions,”
Perhaps. But overall, the possible shortcomings of the Lancet studies persist, in three broad categories.
Design And Implementation Critics say that the surveys used too few clusters, and too few people, to do the job properly.
Sample size. The design for Lancet II committed eight surveyors to visit 50 regional clusters (the number ended up being 47) with each cluster consisting of 40 households. By contrast, in a 2004 survey, the United Nations Development Program used many more questioners to visit 2,200 clusters of 10 houses each. This gave the U.N. investigators greater geographical variety and 10 times as many interviews, and produced a figure of about 24,000 excess deaths — one-quarter the number in the first Lancet study. The Lancet II sample is so small that each violent death recorded translated to 2,000 dead Iraqis overall. The question arises whether the chosen clusters were enough to be truly representative of the entire Iraqi population and therefore a valid data set for extrapolating to nationwide totals. “ ” bias? According to the Lancet II article, surveyors randomly selected a main street within a randomly picked district; “a residential street was then randomly selected from a list of residential streets crossing the main street.” This method pulled the survey teams away from side streets and toward main streets, where car bombs can kill the most people, thus boosting the apparent death rate, according to a critique of the study by Michael Spagat, an economics professor at the Royal Holloway, University of London, and Sean Gourley and Neil Johnson of the physics department at Oxford University.Burnham responds that The Lancet’s description of how the researchers picked sites was an editing error, and that the method used eliminated main-street bias. Oversight. To undertake the first Lancet study, Roberts went into
Main street
With the original data unavailable, other scholars cannot verify the findings, a key test of scientific rigor. Response rate. The surveyors said that 1.7 percent of households — fewer than one in 50 — were unoccupied or uncooperative, even though questioners visited each house only once on one day; that answers were taken only from the household’s husband or wife, not from in-laws or adult children; and that householders had reason to fear that their participation would expose them to threats from armed groups.To Kane, the study’s reported response rate of more than 98 percent “makes no sense,” if only because many male heads of households would be at work or elsewhere during the day and Iraqi women would likely refuse to participate. On the other hand, Kieran J. Healy, a sociologist at the
Death certificates. The survey teams said they confirmed most deaths by examining government-issued death certificates, but they took no photographs of those certificates. “Confirmation of deaths through death certificates is a linchpin for their story,” Spagat told NJ. “But they didn’t record (or won’t provide) information about these death certificates that would make them traceable.”Under pressure from critics, the authors did release a disk of the surveyors’ collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said. For example, the database reveals that 22 death certificates for victims of violence and 23 certificates for other deaths were declared by surveyors and households to be missing or lost. That similarity looks reasonable, but Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single
The data also bolster Spagat’s criticism that the surveyors selected too many clusters in places where bomb explosions and gunfights were most common.
Ideological Bias Virtually everyone connected with the study has been an outspoken opponent of
Follow the money. Lancet II was commissioned and financed by Tirman, the executive director of the Center for International Studies at MIT. (His most recent book is 100 Ways
Partisan considerations. Soros is not the only person associated with the Lancet studies who had one eye on the data and the other on the
“Dr. Burnham and his colleagues are confident that the data presented in the 2004 and 2006 are accurate, and they fully stand by the conclusions of their research,” according to a November 27 statement from the Bloomberg School of Public Health. “The findings of independent surveys of Iraqis conducted by the United Nations in March 2005, by the BBC in March 2007, and by the British polling firm ORB in September 2007 support the conclusions of the
Critics say, however, that the other national reports cited in the Johns Hopkins statement, particularly the ORB poll, have methodological flaws and political overtones similar to those in the Lancet studies.
“Just stating, ‘We have no biases of that type’ isn’t very convincing,” says
Even
The authors — Lafta excepted — have been willing to engage their critics in debate, returning journalists’ calls and, for the most part, avoiding ad hominem arguments. Yet, sometimes their defenses raise new questions. Burnham says, for instance, that Lafta offered to take reporters to visit some of the neighborhoods used in the clusters, although he declined to say whether the reporters would be allowed to visit the surveyed households or to pick the clusters to see.
Roberts and his defenders emphasize that when their cluster method produced shockingly high mortality rates in the
Roberts, when asked if he timed the release of his Lancet studies to hurt the Republicans on Election Day, contends that his biggest concern was ensuring the safety of his researchers. “If this study was finished in September and not published until after the November elections — and it was perceived that we were sitting on the results — my Iraqi colleagues would have been killed,” he told National Journal. Even if true, this assertion undermines his expressions of confidence in the integrity and skill of the Iraqi researchers. How can their data be trusted if their very lives depended on the results?
No matter whether a latent desire to feed the American public’s opposition to the war might have shaped these studies, another audience was paying close attention: jihadists who used this research as a justification for killing Americans. Roberts already believed that jihadi attacks were, in part, driven by the international image of the
Burnham also paused when asked whether Iraqi factions manipulated him and his colleagues and then replied, “We’re reasonably confident that we were not manipulated.”
Professional Responsibilities Officials at Iraq Body Count strongly opposed the
“In the light of such extreme and improbable implications,” the Iraq Body Count report stated, “a rational alternative conclusion to be considered is that the authors have drawn conclusions from unrepresentative data.”
Against these criticisms, the authors maintain that they were using methods of study unfamiliar to human-rights groups and that the scientific community widely accepted the Lancet studies. “There have been 56 studies using this retrospective household survey method,”
When it comes to the question of peer review, the study’s defenders sometimes seem to want it both ways. On the one hand, Roberts talks about the need “to step beyond peer review.” Yet the authors insist that their study was peer-reviewed extensively (if rapidly, in order to be published before the election). The authors also maintain that one of the reasons they went to The Lancet with these studies is its quick turnaround time.
Surprisingly, not one of the peer reviewers seems to have thought to ask a basic question: Are the data in the two studies even true? The possibility of fakery, editor Horton told NJ, “did not come up in peer review.” Medical journals can’t afford to repeat every scientific study, he said, because “if for every paper we published we had to think, ‘Is this fraud?’ … honestly, we would fold tomorrow.”
In
Perhaps medical journals, like respected news organizations, will learn that they have to factor the possibility of wartime fraud into their fact-checking. Horton knows the peacetime risks only too well: In a Lancet article in October 2005, exactly halfway between the two Iraq mortality studies, a Norwegian physician named Jon Sudbo wrote that a review of 454 patients showed that such common painkillers as ibuprofen and naproxen reduced smokers’ risk of contracting oral cancer while increasing their risk for heart disease; it later turned out that Sudbo had faked his research.
Today, the journal’s editor tacitly concedes discomfort with the Iraqi death estimates. “Anything [the authors] can do to strengthen the credibility of the Lancet paper,” Horton told NJ, “would be very welcome.” If clear evidence of misconduct is presented to The Lancet, “we would be happy to go ask the authors and the institution for an official inquiry, and we would then abide by the conclusion of that inquiry.”
Sign up now to get four free issues of National Journal or two free weeks of CongressDaily.
Filed in News | January 31, 2008 | Share This
