By Claire Bernish

A recent examination of National Security Agency documents previously released by whistleblower Edward Snowden shows that the CIA and other U.S. agencies may be killing innocent people as a result of their reliance on metadata.

The NSA’s SKYNET is a program that surveils phone metadata in order to track suspected terrorists. Through SKYNET, the security agency engages in mass surveillance of Pakistan’s mobile phone network, affecting 55 million people — but that’s not all. Once the data is gathered, it’s run through a machine learning algorithm that attempts to rate whether a particular individual is more or less likely to be a terrorist.

According to Human Rights Data Analysis’ executive director Patrick Ball, the NSA’s methods are “ridiculously optimistic” and “completely bullshit.” If Ball is correct, SKYNET’s methodology may be putting thousands of innocent lives in danger because they are being falsely identified as terrorists.

While Ball has raised many issues, one of the most important points, which NSA officials appear to be ignoring, is that the machine is only able to identify a terrorist if it has access to details pertaining to a great number of other known terrorists. According to New York Magazine, however, “there just aren’t that many known terrorists” who could be added to the machine’s list, “especially in comparison to the number of phones the NSA is monitoring in Pakistan.”

The documents made available by The Intercept show the NSA works with the data of only seven known terrorists. NSA officials reportedly feed six of the terrorists’ information into the machine, tasking SKYNET with the duty of finding the seventh in a random group of 100,000 citizens.

Ball says this system cannot work.

According to Ars Technica, there were about 120 million cellular handsets in use in Pakistan at the end of 2012. At the time, the NSA analyzed 55 million of those records. With only “80 variables on 55 million Pakistani mobile phone users, there is obviously far too much data to make sense of manually,” Ars Technica explained.

Like any other application targeting big data, SKYNET is used as a substitute for “human reason and judgment,”Ars Technica’s Christian Grothoff and J.M. Porup explain.

Similar apps, like the one used by Facebook, are prone to making major mistakes, but the consequences of Facebook’s errors are relatively innocuous. When SKYNET makes similar mistakes by wrongly identifying a terrorist, the consequences are deadly.

Ars Technica contends that the information harvested and analyzed by NSA is “likely” used by the CIA or the U.S. military, two agencies that execute “Find-Fix-Finish” strategies with the help of Predator drones or “on-the-ground death squads.”

Before feeding the information to the CIA or the military, SKYNET uses information on people’s typical daily routines to tell a story. According to the NSA documents, the Ars Technica analysis details, the “program … is based on the assumption that the behaviour of terrorists differs significantly from that of ordinary citizens” when it comes to the more than 80 different properties they use to rate people.

With terrorist organizations upping their efforts to remain undetected, it’s hard to see how SKYNET could be effective, especially after learning that Al-Jazeera’s bureau chief in Islamabad, Ahmad Zaidan, was once SKYNET’s highest rated target.

In 2014, the former director of both the CIA and NSA, Michael Hayden, proudly claimed they “kill people based on metadata.”

To Ball, the NSA has a major problem on their hands if they “are using the same records to train the model as they are using to test the model.”

Because there is such a small number of known terrorists to work with, the number of possible terrorists is not sufficiently narrowed down. Ball explains the “usual practice is to hold some of the data out of the training process so that the test includes records the model has never seen before. Without this step, their classification fit assessment is ridiculously optimistic.”

If the NSA is serious about being accurate, Ball told Ars Technica, the agency should mix the terrorists into the population set “before random selection of a subset.” But the low number of known terrorists makes this particular step hard to follow. Without a “scientifically-sound statistical analysis,” there is no accuracy.

Barack Obama has been under heavy scrutiny for authorizing drone campaigns that result in gross mistakes, killing a great number of innocent people while targeting “confirmed terrorists.” Since 2004, there have been 401 US drone strikes in Afghanistan, alone, killing 3,058 people total. In Pakistan, hundreds, including children, have died in pursuit of a mere two dozen declared terrorists.