By Neil Munro and Carl M. Cannon, National Journal

© National Journal Group Inc.

 Friday, Jan. 4, 2008 Three weeks before the 2006 midterm elections gave Democrats control of Congress, a shocking study reported on the number of Iraqis who had died in the ongoing war. It bolstered criticism of President Bush and heightened the waves of dread — here and around the world — about the U.S. occupation of

 Published by The Lancet, a venerable British medical journal, the study [PDF] used previously accepted methods for calculating death rates to estimate the number of “excess” Iraqi deaths after the 2003 invasion at 426,369 to 793,663; the study said the most likely figure was near the middle of that range: 654,965. Almost 92 percent of the dead, the study asserted, were killed by bullets, bombs, or

 In December 2005, Bush had used a figure of 30,000 civilian deaths in

 Queried in the Rose Garden on October 11, the day the Lancet article came out, Bush dismissed it. “I don’t consider it a credible report,” he replied. The Pentagon and top British government officials also rejected the study’s findings. 

Such skepticism would not prove to be the rule. 

CBS News called the report a “new and stunning measure of the havoc the American invasion unleashed in

 Editorials in many major newspapers cited the Lancet article as further evidence that the invasion of

 Within a few weeks a backlash rose, although the contrarian view of the study generated far less press attention than the Lancet article. In the ensuing year, numerous skeptics have identified various weaknesses with the study’s methodology and conclusions. Political blogs and academic journals have registered and responded to the objections in a debate that has been simultaneously arcane and predictable. The arguments are arcane because that is the nature of statistical analysis. They are predictable because that is the nature of today’s polarized political discourse, with liberals defending the Lancet study and conservatives contesting it. 

How to explain the enormous discrepancy between The Lancet’s estimation of Iraqi war deaths and those from studies that used other methodologies? For starters, the authors of the Lancet study followed a model that ensured that even minor components of the data, when extrapolated over the whole population, would yield huge differences in the death toll. Skeptical commentators have highlighted questionable assumptions, implausible data, and ideological leanings among the authors, Gilbert Burnham, Riyadh Lafta, and Les Roberts. 

Some critics go so far as to suggest that the field research on which the study is based may have been performed improperly — or not at all. The key person involved in collecting the data — Lafta, the researcher who assembled the survey teams, deployed them throughout

 Some of these questions could be resolved if other researchers had access to the surveyors’ original field reports and response forms. The authors have released files of collated survey results but not the original survey reports, citing security concerns and the fact that some information was not recorded or preserved in the first place. This was a legitimate problem, and it underscored the difficulty of conducting research in a war zone. 

 Each death recorded by the

Over the past several months, National Journal has examined the 2006 Lancet article, and another [PDF] that some of the same authors published in 2004; probed the problems of estimating wartime mortality rates; and interviewed the authors and their critics. NJ has identified potential problems with the research that fall under three broad headings: 1) possible flaws in the design and execution of the study; 2) a lack of transparency in the data, which has raised suspicions of fraud; and 3) political preferences held by the authors and the funders, which include George Soros’s Open Society Institute.  Origins Of The Survey Since the beginning of the war, the media have meticulously tracked and documented the number of American soldiers killed in Iraq — which reached 3,904 on January 1 — particularly as the total approached and then surpassed (in December 2006) the 2,973 people killed in the 9/11 terrorist attacks. But determining the number of Iraqis who have died is much more difficult, as is determining how many of the dead were insurgents and how many were innocent civilians. With

 The Lancet study was based on techniques developed by public health experts to determine rates of illness and death from epidemics and famines in large populations. This “cluster” sampling is a relatively new methodology that attempts to replicate the logic of public opinion polling in

  Following this method, questioners undertake a house-to-house survey in certain areas and then extrapolate the results from that statistical sample to the entire national population. According to this study’s design, teams of Iraqi questioners would visit approximately 47 randomly chosen clusters of homes throughout the country and ask a series of census-style questions at 40 contiguous households in each cluster: How many people live in your household? How many lived here on January 1, 2002? In that time, how many were born — and how many died? 

In 2004, several of the same authors had done a preliminary

 The Authors The origins of the Lancet studies can be traced to 1993, when two officials from the U.S. Centers for Disease Control and Prevention traveled to Bosnia-Herzegovina to view the devastation caused by the Balkan war. Only nine years after

 In pursuit of an accurate picture, the U.S. health officials toured a hospital in

 “I think that’s when I fully understood the need to step beyond peer-review journals and statistical analyses if you are going to do effective public health work in times of war,” Roberts explained in a recent interview with a Belgian-based publication. This determination to become an advocate would lead him to Rwanda and the Congo, where in 2001 he was involved in studies that produced jaw-dropping estimates of more than 3 million dead in that nation’s civil war. Roberts also went back to the Balkans — this time to Kosovo — and ultimately, when war came to Iraq in 2003, he traveled to

 By then, Roberts was a researcher at the

  A car bomb attack in

 Lafta had been a child-health official in Saddam Hussein’s ministry of health when the ministry was trying to end the international sanctions against Iraq by asserting that many Iraqis were dying from hunger, disease, or cancer caused by spent U.S. depleted-uranium shells remaining from the 1991 Persian Gulf War. In 2000, Lafta authored at least two brief articles contending that U.N. sanctions had caused many deaths by starvation among Iraqi children. In one article, he identified malnutrition as the main contributor to 53 percent of deaths among hospitalized children younger than 2, during a 1997 survey carried out at Saddam Central Teaching Hospital. The article cited no health data from before the sanctions, yet it asserted, “We can conclude from results that the most important and widespread underlying cause of the deterioration of child-health standards in Iraq is the long-term impact of the nonhumanized economic sanction imposed through United Nations resolutions.” The article was published in 2000 by the Iraqi Journal of Community Medicine. Roberts told National Journal he had not read Lafta’s articles, and Burnham said he did not have a copy of the articles.  

Lafta is now at

 Lafta and his surveyors often worked under brutal political pressure. In January 2007, a Sunni suicide bomber killed more than 70 students at the university, partly because it is perceived as being under the control of Moktada al-Sadr, the Shiite religious leader whose Mahdi Army militia crippled Sunni insurgent groups in

 Dramatic Findings In his first study of Iraqi war deaths, in September 2004, Lafta sent six Iraqi questioners to 33 clusters of homes throughout the country to ask how many people in each household had died since January 1, 2002. The researchers reported that 808 of the 998 identified households participated in the survey, and then extrapolated the number of deaths reported to the entire population of 24.4 million Iraqis. “Making conservative assumptions, we think that about 100,000 excess deaths or more have happened since the 2003 invasion of Iraq,” concluded the authors — Roberts, Lafta, Garfield, Jamal Khudhairi, and Burnham. That was when the war was just 19 months old. 

“Violence accounted for most of the excess deaths, and air strikes from coalition forces accounted for most violent deaths,” the report said. According to subsequent explanations by the authors, the total included 57,600 dead from violence, 24,000 dead from wartime accidents, and 13,600 dead from disease. The accidental deaths included 15,000 Iraqis killed by

 Little is known about Lafta’s decision-making in amassing the data for the Lancet surveys. Roberts provided some information, however, about Lafta’s 2004 survey of casualties in Falluja. At the time, al-Sadr was publicly supporting the anti-American Sunni radicals who controlled the city. In September, Roberts said, he pleaded with “his Muslim friend Lafta not to go” into Falluja, according to an interview with a magazine published by Johns Hopkins. Roberts told the interviewer that Lafta replied, “God has picked these clusters. If God wants me, he will take me. I must go.” Roberts also said of Lafta, “I know no one [who] perceives themselves so humbly to be a tool of God’s destiny…. He sees his science as synonymous with service to God.” 

In Falluja, Lafta recorded 52 deaths in 29 households, which amounted to 71 percent of the violent deaths recorded by the first Lancet survey. If representative, Lafta’s sample translated into 50,000 to 70,000 dead in Falluja by September 2004 — two months before the start of the second major American military operation to restore order. Falluja’s prewar population was estimated to be 250,000, although

 The 2006 study, known as Lancet II, was somewhat larger, involving 47 clusters and using similar survey techniques. In all, 302 violent deaths reported in those 1,849 households became the basis for estimating that 601,000 Iraqis had died violently from the start of the war through June 2006. 

Even though the second study was even further out of line with other sources’ estimates than the first, it got tremendous attention — probably because its findings fit an emerging narrative:

 Democrats who had opposed Bush’s

 The Lancet II article was also publicized widely overseas, especially in the

 Muslim commentators in the

 In the

  The John Hopkins researcher, Les Roberts, began the studies by smuggling himself into

 The study had such a significant impact partly because of where it appeared. The Lancet, founded in 1823, is one of the world’s most-cited medical journals, credited with publishing articles that established the principles of antiseptics in 1867 and documented the dangers of thalidomide in 1961. Although few mainstream journalists ever plow through the journal’s articles, news outlets typically refer to it as “the respected Lancet.” In recent years, however, the journal’s reputation has suffered from charges of politicization and a few prominent instances of scientific fraud. 

Also driving the press attention was the study’s association with Johns Hopkins University, whose School of Public Health was the first and is now the largest such institution in the world. Faculty members participated in the study, and the school’s review board conducted an ethical review of the research plan. The Arab American’s Siblani said that the university connection was one reason he put the study on the front page of his newspaper. 

Potential Problems Both Lancet studies of Iraqi war deaths rest on the data provided by Lafta, who operated with little American supervision and has rarely appeared in public or been interviewed about his role. In May, Lafta and Roberts presented their study to an off-the-record meeting of experts in

 When asked questions about the reliability of their Iraqi partner, the studies’ American authors defend Lafta as a nice guy and a good researcher. 

“I’ve known him for years,”

 John Tirman, a political scientist at the Massachusetts Institute of Technology, described Lafta as “a medical doctor, a professor of medicine. Those factors were a sufficient level of credibility. I never asked [Lafta] about his political views.” Tirman commissioned the Lancet II survey with $46,000 from George Soros’s Open Society Institute and additional support from other funders. 

Lancet Editor Richard Horton shares this fundamental faith in scientists. He told NJ that scientists, including Lafta, can be trusted because “science is a global culture that operates by a set of norms and standards that are truly international, that do not vary by culture or religion. That’s one of the beautiful aspects of science — it unifies cultures, not divides them.” 

Still, the authors have declined to provide the surveyors’ reports and forms that might bolster confidence in their findings. Customary scientific practice holds that an experiment must be transparent — and repeatable — to win credence. Submitting to that scientific method, the authors would make the unvarnished data available for inspection by other researchers. Because they did not do this, citing concerns about the security of the questioners and respondents, critics have raised the most basic question about this research: Was it verifiably undertaken as described in the two Lancet articles? 

“The authors refuse to provide anyone with the underlying data,” said David Kane, a statistician and a fellow at the Institute for Quantitative Social Statistics at

 Fritz Scheuren, vice president for statistics at the National Opinion Research Center and a past president of the American Statistical Association, said, “They failed to do any of the [routine] things to prevent fabrication.” The weakest part of the Lancet surveys is their reliance on an unsupervised Iraqi survey team, contended Scheuren, who has recently trained survey workers in

  When the study came out in October 2006, President Bush said it wasn’t credible.    

The research is “a field study in unstable conditions,”

 Perhaps. But overall, the possible shortcomings of the Lancet studies persist, in three broad categories. 

Design And Implementation Critics say that the surveys used too few clusters, and too few people, to do the job properly. 

Sample size. The design for Lancet II committed eight surveyors to visit 50 regional clusters (the number ended up being 47) with each cluster consisting of 40 households. By contrast, in a 2004 survey, the United Nations Development Program used many more questioners to visit 2,200 clusters of 10 houses each. This gave the U.N. investigators greater geographical variety and 10 times as many interviews, and produced a figure of about 24,000 excess deaths — one-quarter the number in the first Lancet study. The Lancet II sample is so small that each violent death recorded translated to 2,000 dead Iraqis overall. The question arises whether the chosen clusters were enough to be truly representative of the entire Iraqi population and therefore a valid data set for extrapolating to nationwide totals. “

With the original data unavailable, other scholars cannot verify the findings, a key test of scientific rigor. Response rate. The surveyors said that 1.7 percent of households — fewer than one in 50 — were unoccupied or uncooperative, even though questioners visited each house only once on one day; that answers were taken only from the household’s husband or wife, not from in-laws or adult children; and that householders had reason to fear that their participation would expose them to threats from armed groups.To Kane, the study’s reported response rate of more than 98 percent “makes no sense,” if only because many male heads of households would be at work or elsewhere during the day and Iraqi women would likely refuse to participate. On the other hand, Kieran J. Healy, a sociologist at the

Death certificates. The survey teams said they confirmed most deaths by examining government-issued death certificates, but they took no photographs of those certificates. “Confirmation of deaths through death certificates is a linchpin for their story,” Spagat told NJ. “But they didn’t record (or won’t provide) information about these death certificates that would make them traceable.”Under pressure from critics, the authors did release a disk of the surveyors’ collated data, including tables showing how often the survey teams said they requested to see, and saw, the death certificates. But those tables are suspicious, in part, because they show data-heaping, critics said. For example, the database reveals that 22 death certificates for victims of violence and 23 certificates for other deaths were declared by surveyors and households to be missing or lost. That similarity looks reasonable, but Spagat noticed that the 23 missing certificates for nonviolent deaths were distributed throughout eight of the 16 surveyed provinces, while all 22 missing certificates for violent deaths were inexplicably heaped in the single

The data also bolster Spagat’s criticism that the surveyors selected too many clusters in places where bomb explosions and gunfights were most common. 

Ideological Bias Virtually everyone connected with the study has been an outspoken opponent of U.S. actions in

 Follow the money. Lancet II was commissioned and financed by Tirman, the executive director of the Center for International Studies at MIT. (His most recent book is 100 Ways

Partisan considerations. Soros is not the only person associated with the Lancet studies who had one eye on the data and the other on the

“Dr. Burnham and his colleagues are confident that the data presented in the 2004 and 2006 are accurate, and they fully stand by the conclusions of their research,” according to a November 27 statement from the Bloomberg School of Public Health. “The findings of independent surveys of Iraqis conducted by the United Nations in March 2005, by the BBC in March 2007, and by the British polling firm ORB in September 2007 support the conclusions of the

 Critics say, however, that the other national reports cited in the Johns Hopkins statement, particularly the ORB poll, have methodological flaws and political overtones similar to those in the Lancet studies.  

“Just stating, ‘We have no biases of that type’ isn’t very convincing,” says

 Even

 The authors — Lafta excepted — have been willing to engage their critics in debate, returning journalists’ calls and, for the most part, avoiding ad hominem arguments. Yet, sometimes their defenses raise new questions. Burnham says, for instance, that Lafta offered to take reporters to visit some of the neighborhoods used in the clusters, although he declined to say whether the reporters would be allowed to visit the surveyed households or to pick the clusters to see. 

Roberts and his defenders emphasize that when their cluster method produced shockingly high mortality rates in the Congo, no one questioned them — not seeming to understand that journalists looking at the Iraq study are now indeed wondering if the

 Roberts, when asked if he timed the release of his Lancet studies to hurt the Republicans on Election Day, contends that his biggest concern was ensuring the safety of his researchers. “If this study was finished in September and not published until after the November elections — and it was perceived that we were sitting on the results — my Iraqi colleagues would have been killed,” he told National Journal. Even if true, this assertion undermines his expressions of confidence in the integrity and skill of the Iraqi researchers. How can their data be trusted if their very lives depended on the results? 

No matter whether a latent desire to feed the American public’s opposition to the war might have shaped these studies, another audience was paying close attention: jihadists who used this research as a justification for killing Americans. Roberts already believed that jihadi attacks were, in part, driven by the international image of the

 Burnham also paused when asked whether Iraqi factions manipulated him and his colleagues and then replied, “We’re reasonably confident that we were not manipulated.” 

Professional Responsibilities Officials at Iraq Body Count strongly opposed the

 “In the light of such extreme and improbable implications,” the Iraq Body Count report stated, “a rational alternative conclusion to be considered is that the authors have drawn conclusions from unrepresentative data.” 

Against these criticisms, the authors maintain that they were using methods of study unfamiliar to human-rights groups and that the scientific community widely accepted the Lancet studies. “There have been 56 studies using this retrospective household survey method,”

 When it comes to the question of peer review, the study’s defenders sometimes seem to want it both ways. On the one hand, Roberts talks about the need “to step beyond peer review.” Yet the authors insist that their study was peer-reviewed extensively (if rapidly, in order to be published before the election). The authors also maintain that one of the reasons they went to The Lancet with these studies is its quick turnaround time. 

Surprisingly, not one of the peer reviewers seems to have thought to ask a basic question: Are the data in the two studies even true? The possibility of fakery, editor Horton told NJ, “did not come up in peer review.” Medical journals can’t afford to repeat every scientific study, he said, because “if for every paper we published we had to think, ‘Is this fraud?’ … honestly, we would fold tomorrow.” 

In

 Perhaps medical journals, like respected news organizations, will learn that they have to factor the possibility of wartime fraud into their fact-checking. Horton knows the peacetime risks only too well: In a Lancet article in October 2005, exactly halfway between the two Iraq mortality studies, a Norwegian physician named Jon Sudbo wrote that a review of 454 patients showed that such common painkillers as ibuprofen and naproxen reduced smokers’ risk of contracting oral cancer while increasing their risk for heart disease; it later turned out that Sudbo had faked his research. 

Today, the journal’s editor tacitly concedes discomfort with the Iraqi death estimates. “Anything [the authors] can do to strengthen the credibility of the Lancet paper,” Horton told NJ, “would be very welcome.” If clear evidence of misconduct is presented to The Lancet, “we would be happy to go ask the authors and the institution for an official inquiry, and we would then abide by the conclusion of that inquiry.” 

Sign up now to get four free issues of National Journal or two free weeks of CongressDaily.  

Filed in News | January 31, 2008 | Share This  

The Next Iraq Phase
By David Ignatius
Wednesday, January 30, 2008; A15

BAGHDAD — America’s future role in Iraq is being shaped by two discussions underway here and in Washington. One is a Bush administration debate about the timetable for reducing U.S. troops this year, and the other is a U.S.-Iraqi negotiation about the status of the residual American force that will remain after 2008.

The premise of these discussions is that U.S. policy in Iraq is finally working and that a framework must be found to preserve the security gains of the past year. But this military planning fits awkwardly with the political mood in the United States and Iraq — where the publics remain skeptical about U.S. military occupation, even when it’s finally achieving its goals.

Progress here is undeniable, both in terms of security on the ground and in the political bargaining among Iraq’s parties and ethnic groups. You see this on the streets, in the faces of people you meet in shops and teahouses. The Iraqis I met last weekend didn’t complain about security but about delivery of services. There are also hints of pragmatism among Iraqi politicians, who are finally passing legislation after three years of political deadlock.

The question is whether this Iraqi renaissance can continue as the United States reduces its surge of combat troops. The Iraqi military is still far from ready to take over the country’s security. The military’s transport systems won’t be finished until the summer of 2009, and it could be two years before Iraq’s military can operate fully independent of U.S. forces.

Gen. David Petraeus and other top military officials have begun debating what the post-surge level of U.S. troops should be. The commanders want a pause for assessment after July, when the last of the five additional combat brigades that made up the surge is withdrawn and the U.S. troop presence returns to its prior level of 15 brigades, or about 130,000 soldiers.

The debate centers on how long this pause should last and whether it should be followed by more troop cuts. Petraeus, who as field commander doesn’t want to risk losing his hard-won gains, is said to favor an assessment period of more than three months, and perhaps leaving the full 15 brigades in place through the end of 2008. President Bush, who would like to leave office next January with Iraq as secure as possible, may also oppose further troop reductions after July.

A contrary view favors a continuing process of withdrawals that is politically sustainable in both Iraq and America. Advocates of this view, who include some top Pentagon and Central Command officials, worry that maintaining troops at the full, pre-surge level of about 130,000 will stress the U.S. Army to the breaking point.

There’s also the political risk factor: If Bush maintains a large force until Inauguration Day 2009, the next president might order a drastic switch in policy — with big, sudden troop cuts. That could have a disastrous effect on Iraqi security. Thus, some Pentagon and Centcom officials have argued for a steadier glide path of troop reductions that would allow realistic planning by the U.S. and Iraqi militaries for operations in 2009 and beyond.

The U.S. debate will come to a head when Petraeus and U.S. Ambassador Ryan Crocker give their next progress report to Congress, probably in early April.

Meanwhile, U.S. officials are beginning to negotiate with the Iraqis the legal rules under which U.S. forces will operate in 2009 and beyond. The goal is a new “Security Framework Agreement” to replace the current U.N. mandate that expires at the end of this year. Among the tricky issues are the legal authorities for U.S. troops to conduct operations against al-Qaeda and Iranian-backed militias; permission for the United States to continue holding Iraqi detainees, who currently number about 24,000; and U.S. rights to operate military bases in Iraq. The Iraqis are planning to hire U.S. lawyers who have experience negotiating such “status of forces agreements.”

All this planning sounds sensible enough when you listen to top U.S. and Iraqi officials here. Their goal is a stable Iraqi state, which is less a pipe dream now than it was a year ago.

The problem is that the security discussions are taking place against a political backdrop of impatient, war-weary Iraqis and Americans. The Iraqis want a restoration of full sovereignty, and they aren’t likely to tolerate for much longer the American-run prisons or U.S. soldiers kicking down doors.

Unless the planners take that political reality into account — and reassure Iraqis and Americans alike that most U.S. troops will gradually be coming home — they may be creating a new version of Mission Impossible.

The writer is co-host ofPostGlobal, an online discussion of international issues. His e-mail address isdavidignatius@washpost.com.

Man of the DecadeThanks to Gen. Petraeus, the Iraq War has gone from probable loss to likely successby : Josh Levy

With fewer stories about

The Fixer: Under Gen. Petraeus’ (left) watch, more Iraqis have begun cooperating with the

In 2003, after Saddam’s swift defeat and overthrow,

For nearly three years, we failed to change our strategy to meet this challenge, and by early 2006,

Meanwhile in

Remarkably, the enemies in

Congress could not ignore this progress, and the momentum to retreat eventually died away. Now it is so common for congressmen who once wanted to stop fighting to begin supporting the

Victory has not yet arrived, and it may be years before we can mark its arrival with confidence, but we can reasonably hope to see it. 

Time magazine was correct not to name Gen. Petraeus “Man of the Year.” He is “Man of the Decade.” 

Josh Levy (peacethruvictory@gmail.com) led a pro-victory rally against Cindy Sheehan last July.

Many of America’s most decorated war heroes from Iraq and Afghanistan have packed their bags and are hitting the road on a national bus tour to take their non-partisan message of progress and freedom from coast-to-coast.

We hope see you along the way—come out, meet some American heroes and support our troops!

The National Heroes Tour
(updated as of January 30, 2008—subject to updates)

A familiar sight in Iraq

by Chuck Yarborough

Wednesday January 09, 2008, 7:57 AM

Marine Maj. Bob ReynoldsMarine Maj. Bob Reynolds was accompanying an Iraqi army patrol in an area between Ramadi and Samarra when he spotted this boy in a Cleveland Cavaliers shirt. The boy told Reynolds he got the shirt from a cousin.

Athletic jerseys are as common on the streets of Iraq as they are in Ohio. Only most of the time, the kids are wearing soccer jerseys from European and Middle Eastern countries. That’s why this youngster caught the eye of Marine Maj. Bob Reynolds, an intelligence officer attached to the Iraqi army as an adviser. It’s not too often you see a Cleveland Cavaliers polo shirt on the dusty roadways of Samarra.Reynolds, a native of Lakewood and former Strongsville police officer, is an adviser to the Iraqi unit conducting operations in the eastern reaches of Iraq’s Al Anbar province. This is his second deployment to Iraq. He is scheduled to return to the United States later this month. He’s based at Camp Lejeune, N.C.