An October report by Columbia Law School's Human Rights Clinic claims to have found significant flaws in media reports regarding casualties caused by U.S. drone operations in Pakistan. Three organizations, the Bureau of Investigative Journalism, the Long War Journal and the New America Foundation, maintain databases that collect information on the casualties for each strike and their research is regularly cited in congressional reports and news articles. While the Columbia report laments that these estimates can only "substitute for hard facts and information that ought to be provided by the U.S. government," it proceeds to weigh in on the casualty debate. After a strike by strike comparison of the three databases' 2011 data, Columbia concludes that two of these organizations "significantly undercount the number of civilians killed by drone strikes," while singling out the Bureau as the most accurate and reliable source of information on drone casualties.

The Columbia study is quick to critique the drone data compiled by the New America Foundation and the Long War Journal, yet it devotes negligible attention to potential shortcomings in the Bureau's reporting. The study repeatedly applauds the Bureau's investigative practices, analysis criteria and the breadth of its sources. It offers the most guarded criticism, writing only, "we do not agree with the Bureau's analysis of media sources in all cases." Upon reading Columbia's "Counting Drone Strike Deaths," one is led to conclude that the Bureau's casualty estimates are both methodologically rigorous and empirically sound.

And yet, a careful reading of the separate 65 page dataset, which details the findings of Columbia's exhaustive comparison, reveals numerous instances in which the Columbia researchers reject the Bureau's interpretation of the evidence or dispute the credibility of their sources, criticisms that receive no mention in their widely circulated report.

• Regarding a strike on January 7, 2011, the Bureau reports 0-6 civilians dead, citing the Pakistan Observer's claim that the dead were "ordinary tribesman." However, according to Columbia's analysis, "No identifying information is provided supporting either the claim that those killed were militants or that they were civilians."

• Analysis of a February 20, 2011 strike reveals the danger of counting civilian deaths based on a single source. Columbia writes, "We believe that The News' report that all the dead were tribal people is not clear enough to suggest that they were civilians, in light of the contrary reports." Based on this same evidence the Bureau lists 0-11 civilian deaths.

• The conclusions drawn from a strike the following day further demonstrate the inconclusive nature of the media reporting. "Anonymous officials' statements are corroborated by locals' reports of a Taliban base, as against The News report that villagers said those killed were not affiliated with militants. There is no further identifying information to support either claim."

• Columbia's researchers also make note of the Bureau's expansive interpretation of certain news reports. Regarding a strike on July 11, 2011, none of the ten sources referenced by the Bureau cite civilian casualties. One report says the identity of the victims is unknown and the Bureau lists 0-5 civilian casualties. According to Columbia, "In our view, reports indicating that the identity of the deceased is not known are not sufficient to amount to reports of civilian deaths."

Columbia only analyzed reports for 2011, but had they continued on with their research, they would have found that these problems pervade the Bureau's reporting on strikes from 2004 through 2012.

• During this period there are thirteen strikes for which the Bureau lists civilian deaths based on reports that the dead were "locals." This accounts for 1-80 of their reported civilian casualties.

• Sixteen of the strikes for which they list civilian deaths are only corroborated by a single media report, and yet these represent 44-94 of the alleged civilian deaths.

• There are at least 13 strikes where the sources make no reference to civilian deaths or locals and yet the Bureau lists zero to 66 civilian casualties. In these instances the dead were either listed as "people" or the Bureau seems to have inferred that any casualty not reported specifically as a militant was a civilian.

Based on this tenuous evidence the Bureau has claimed 45-240 civilian casualties. Taken together, this methodologically flawed reporting accounts for over 25 percent of the 474-884 civilian casualties the Bureau claims died between 2004 and 2012. While it is highly probable that some of these deaths may in fact have been civilians, in the face of so much ambiguity, it would be more prudent to label the deceased as unidentified or unknown. This would provide a more accurate representation of the evidence, and acknowledge that despite the best attempts to gather information, there is still much uncertainty about the outcome of individual strikes and the overall effect of the U.S. drone program.

The trends highlighted above point to three broader methodological flaws in the Bureau's analysis that Columbia researchers fail to highlight.  The first is a problem of evidence. The Columbia report suggests that the widest range of sources will provide the most credible evidence, and based on the fact that the Bureau cites the largest body of sources and has the highest casualty figures, its numbers are the most reliable. Beyond the fact that the Bureau is a notable outlier as compared to the other two datasets produced by New America Foundation and The Long War Journal, it is a mistake to privilege quantity of evidence over quality. Pakistan Body Count, South Asia Terrorism Portal and the Long War Journal are secondary sources that rely on reporting from other media outlets and should not count as corroborating sources, yet the Bureau does so. Antiwar.com, sify.com, Prison Planet and the World Socialist Web Site are simply not credible news outlets, yet these are some of the sources that the Bureau is praised for citing.

The second problem is the absence of transparency in investigations of drone strikes carried out by the Bureau's own researchers. The Bureau says it has conducted independent investigations of certain strikes and their database includes 13 strikes where the sole source of information citing civilian deaths is from "the Bureau's own researchers." These uncorroborated claims account for at least 56-64 of the Bureau's reported civilian casualties. The strike on January 6, 2010 includes a typical description: "According to the Bureau's researchers five rescuers died, named as Khalid, Matiullah, Kashif, Zaman and Waqar, all belonging to the Utmanzai Wazir tribe." However there is no indication of whom these researchers are or the standards they apply to their reporting.

The same criticisms that the Columbia report levies at unnamed Pakistani government officials discussing drone strikes on the condition of anonymity, could just as easily be aimed at the Bureau's own reporting:

We do not know who the unnamed Pakistani officials are, although observers believe they are Pakistani army officials. What definition these officials use to categorize a person as a militant or civilian is unknown. Nor do we know how the Pakistani Army confirms such deaths or the quality of information it is able to rely on given the limited accessibility of some of the tribal regions to even the Army.

The Bureau's researchers might well be the sort of local journalists or "stringers" the Columbia report is quick to term unreliable. Nor does the Bureau make mention of whom their sources are, when or where they were interviewed, or what was said. If the Bureau wants its findings to be taken seriously by other researchers, then it should provide independent reports of its investigations rather than cursory references in the midst of its dataset.

The third problem is one of interpretation. The Bureau consistently counts references to "local" deaths as civilian casualties, but as the Columbia dataset notes, these descriptors are not synonymous. The media reports are riddled with references to "local militants" and "tribal militants." It stands to reason that a significant number of the militants operating in the tribal regions of Pakistan would live in the area, and thus the mere fact that the deceased are reported as local is not sufficient to establish that they are civilians. And yet, the Bureau consistently claims just that. Even worse, it frequently labels the fatalities as civilians when the media accounts refer to them in neutral terms such as "people" or admits that their identity is unclear.

Furthermore, the Bureau's written methodology provides limited insight into how it makes these interpretations. The methodology makes no mention of how the Bureau treats reports of "local" deaths. Nor does it explain how it deals with conflicting reports. The methodology says that when reports differ it provides a range of total casualties, but it does not explain how the Bureau determines who to count as a civilian. It goes on to state that "where media sources refer only to ‘people' killed... we indicate that civilian casualties may be possible." One would assume they indicate this by way of an asterisk or note, but it would seem that in most instances it reports a range of civilian casualties with a low end of zero and a high end of the total killed. This denotes the uncertainty but potentially inflates the high end of the range of civilian deaths. Moreover, it signals a clear preference for counting unidentified casualties as civilians.

Admittedly, this somewhat esoteric discussion about the veracity of the Bureau's claims versus those of other databases, or the appropriate methodology for counting casualties, risks losing sight of the broader picture. These are not merely numbers; these are people. And no matter which database you reference, civilians are being killed by the hundreds. While this consideration should be paramount, an assessment of the drone program should also take into account those factors that are less quantifiable: the elevated rates of PTSD in areas where drones operate, the dangerous example being set for other states, most notably China and Russia, and the increase in anti-U.S. sentiment in Pakistan that risks endangering the lives of American citizens.

But as the Columbia study points out, numbers matter. Numbers drive our public discourse. Numbers are how politicians measure outcomes. Numbers are how we make sense of our world. And numbers are vulnerable to manipulation, a distortion that is equally dangerous whether it involves government officials lowballing civilian casualty reports or independent researchers potentially inflating them.

Meg Braun is a Rhodes Scholar and MPhil candidate in International Relations at Oxford, where she is researching the evolution of U.S. drone policy. She was an intern at the New America Foundation during summer 2012, where she worked to revise and update its drone database.

Correction: This post initially stated incorrectly that "The Bureau says it has conducted independent investigations of certain strikes and their database includes 15 strikes where the sole source of information citing civilian deaths is from "the Bureau's own researchers." These uncorroborated claims account for at least 65-73 of the Bureau's reported civilian casualties." The correct numbers are 13 strikes and 56-64 reported civilian casualties.

Though the embattled Pakistan Peoples Party (PPP)-led government breathed a sigh of relief after passing what may be called a "pro-democracy" resolution in parliament on the evening of January 16, hours later the country's Supreme Court issued a contempt of court notice to the Prime Minister Yousaf Raza Gilani for his refusal to reinstate corruption charges filed against President Asif Ali Zardari in a Swiss court. 

Many analysts see the political crisis currently wracking Pakistan as a do-or-die moment for its civilian government. However, the country's all-powerful army is also feeling the heat of events this time, mainly due to the visible shift in public opinion against an explicit military intervention in the country's politics.

Despite provocative, albeit well-placed, ‘state-within-the-state' comments by Prime Minister Yousaf Raza Gilani about the army and its powerful intelligence arm the Inter-Services Intelligence Directorate (ISI), or Prime Minister Gilani's decision to sack the well-respected Defense Secretary Naeem Khalid Lodhi, the military leadership so far opted to keep its hands off direct intervention, instead opting to voice its displeasure in the media.

Instead, analysts believe, the generals are using their mighty arm behind the scene, by pushing an interventionist Supreme Court not to let the civilian government off the hook.

The first case in question is the murky memo addressed to former U.S. Chairman of the Joint Chiefs of Staff Adm. Mike Mullen allegedly asking for help stopping the Pakistani generals from carrying out a coup following the May 2 raid in Abbottabad. The second is the controversial National Reconciliation Ordinance (NRO) introduced by former dictator Pervez Musharraf, which grants amnesty to all political leaders, workers and bureaucrats accused of corruption, embezzlement and misuse of authority between January 1, 1986 and October 12, 1999.

Notwithstanding the media criticism of the civilian government over a host of issues including good governance, the country's poor economic situation, law and order problems in places like Karachi, Pakistan's seething energy crisis and apologetic approach towards militancy, the majority of leading analysts, newspapers and television commentators have come out clearly against extra-constitutional measures and an overt seizure of power.

In his commentary in the U.K.-based Guardian newspaper, journalist Muhammad Hanif says the Pakistan army is at least partially responsible for the troubles afflicting the nuclear-armed country and its 180 million people: "Pakistan's army is as corrupt as the politicians from whom it wants to save the country. It's just better at paperwork."

During the past three major coups in 1958 (Gen. Ayub Khan), 1977 (Gen. Zia) and 1999 (Gen. Musharraf), political leaders, civil society and even the majority of media outlets welcomed the change, hoping for a better future for the country. However, there are no such feelings visible this time, not even from the staunchest opponents of the government among politicians, civil society and the media.

Following the Army's prediction of "grievous consequences" in response to Prime Minister Gilani's interview with a Chinese newspaper, one of Pakistan's leading newspapers, Express Tribune, put a key question before its readers about the army's role in the country's politics: "The first question that comes to mind as one reads this is, did the military's actions in 1958, 1977 and 1999 also reflect an "allegiance to State and the Constitution"? Is not a former army chief on record as having said that the Constitution was a mere piece of paper?"

Discussing the same subject, another leading newspaper, Dawn writes: "One thing in particular bears stating: if Pakistan had been a more developed democracy, the authors of the ISPR [Inter-Services Public Relations] statement this week would have been summarily sacked." 

Additionally, many journalists in Pakistan seem to be more aware of their critical role in saving democratic institutions this time. In her article in Express Tribune, analyst Nasim Zehra writes: "Had there been an independent electronic media in October 1999 there would have been no coup."

What is different now, though, more than three years after Pakistan's return to democracy, is the role played by the Supreme Court as a perceived advocate of the armed forces. Discussing the recent decision of the Supreme Court questioning the ‘honesty' of Prime Minister Gilani on the basis of Quranic injunctions against being deceitful, a Daily Times columnist Dr. Muhammad Taqi writes: "In a country reeling under the effects of radicalization, the last thing needed is the industrial-strength moral certitude and virtual proselytizing from the bench." In his article entitled "Judicial Hubris," Dr. Taqi states that "it is most unfortunate that the honorable judges have repeatedly resorted to religious rhetoric to establish the case against the NRO beneficiaries."

Another columnist, Kamran Shafi, writing in Express Tribune on the same subject, asks the Supreme Court as why the judicial commissions are silent over the role of intelligence agencies in their failure to track down bin Ladin in garrison town of Abbottabad or the culprits behind the tragic murder of journalist Saleem Shahzad. Shafi continues: "What is of utmost import today; what is a matter of life or death for many Pakistanis; what will determine whether we are a civilized people or a horde of wild brutes is the shamefully non-conclusive report on the brutal and savage beating to death of journalist Saleem Shahzad."

Like several other analysts, Ilyas Khan of the BBC believes that the army is supporting the Supreme Court behind the scene to push the government to the corner. "Instead, the military are thought to prefer to let the Supreme Court use "constitutional" methods to go after the government."

Meanwhile, the Urdu-language newspapers, mostly known for their anti-American and anti-government comments, have generally continued to criticize the government's inefficiency, but have still asked for an end to the crisis in accordance with the tenets laid out in the Constitution of Pakistan.

In its editorial on January 15, just a day after the government introduced a pro-democracy resolution in the parliament the Urdu-language Daily Express praised Pakistan's political parties for struggling to resolve the crisis through democratic means.

Another Urdu-language newspaper, the Daily Mashriq, criticizes the government for its ‘inefficiency' and ‘non-implementation' of the court decisions regarding the NRO, but also opposes the tussle among the state institutions (i.e. the parliament, army and judiciary), arguing that this infighting will have negative effects on the future of democracy.

Commentator and analyst Ayaz Amir, in his article entitle "double standards and hypocrisy" in the Urdu daily Jang, says no one can deny the fact that the present government is inefficient. But, he says, it is time for the opposition to let it complete the remaining one year on its term for the sake of democracy.

***

The rapid pace of movement on the political front makes predictions impossible, though the most likely scenario will be the government's agreement with the opposition parties, both inside and outside the parliament, to an early election following the voting for Senate, due to take place in March of this year.

While the military is in no position to stage a direct coup for a number of reasons, including opposition from both pro- and anti-government parties alike, it is the Supreme Court of Pakistan that poses the most direct threat to the existing government serving out its current term.

The opposition parties have their own axes to grind. The main opposition party, the Pakistan Muslim League-Nawaz (PML-N) is apparently resisting an overt army action, but will not weep if the government is sacked by the Supreme Court with behind-the-scenes approval from the army.

The reason for allowing this to happen is clear: To stop the PPP from getting a majority in the Senate election due in March and do not allow more time to cricketer turned politician Imran Khan, whose previously downtrodden Tehrik-e-Insaaf is unexpectedly making headway in many cities. Khan's critics believe he enjoys secret support from "the establishment," which means the army and its intelligence agencies. 

The people of Pakistan, suffering under price hikes for energy and gasoline, high unemployment, and numerous other problems, would shed no tears if the government were sent packing under pressure from the Supreme Court. However, a direct army intervention is likely to be resisted, mainly because of the army's shattered image following the Musharraf era.

Some sources in the pro-PPP camp say the government would rather to be removed through direct army intervention than by the Supreme Court, just to become a ‘Siasee Shaheed' (political martyr) and garner public sympathy before the next general elections. Hence, political circles close to the PPP leadership may not rule out steps provoking the army -- including seeking the resignation or sacking of the Army and ISI chiefs -- once the party sees clear chances of removal from government through the Supreme Court. One last option for the government, in a bid to avoid the Supreme Court action, is the resignation of Prime Minister Gilani, which could postpone, if not fully avert, the existing crisis -- until the crucial Senate election at least. Prime Minister Gilani is due to appear before the Supreme Court on Thursday, January 19 with regard to the NRO case.

With nothing clear about the future, the only solid element seen on Pakistan's political horizon is the strong resolve shown by the people, media, civil society and the political parties to say "no" to a possible military intervention and ‘yes' to democracy and to the supremacy of the Constitution. What that will mean in reality, though, is anyone's guess.

Daud Khattak is a journalist currently working for the Radio Free Europe/Radio Liberty's Pashto-language station Radio Mashaal.