This past May, at a London summit hosted by the UK’s Royal Aeronautical Society, the U.S. Air Force’s chief of AI test and operations brought up an unnerving scenario. He told the audience about a simulation involving an AI-enabled drone tasked with identifying and destroying targets. A human operator retained the final authority over whether to ultimately destroy any one target.
The AI drone soon realized that its human operator, with their pesky ability to make decisions that occasionally resulted in a call to not destroy a target, stood in the way of the drone’s given task. To better meet its directed objective— to find and destroy targets — the drone killed the operator.
The internet lit up. Social media was awash in references to Skynet, the military AI system from the Terminator franchise, which becomes sentient and tries to stamp out humanity.
The Air Force quickly said it had never run through such a scenario: The situation was merely a thought exercise. In updated comments to the Royal Aeronautical Society, the officer, Col. Tucker Hamilton, wrote, “We’ve never run that experiment.”
But his comments didn’t preclude the possibility of such a scenario: “(Nor) would we need to in order to realize that this is a plausible outcome,” he continued.
. . .
The military has been investing in and benefiting from artificial intelligence for years — from unmanned aerial systems to autonomous weapons systems to data and image processing technologies. In August, the military announced it was installing a new AI-enabled monitoring and warning system to help protect the airspace over Washington.
c It has also requested an additional $1.4 billion for developing and testing its Joint All-Domain Command and Control initiatives, which aim to create an AI-powered network of connected sensors between military assets across all branches.
The pace of developing artificial intelligence systems in the civilian world has accelerated drastically in recent years, often led by small, nimble companies, many of which are releasing technologies into the world whose far-reaching implications are not yet known. Applications like GPT-4 have opened the public’s eyes to just how world-changing these technologies might be, further increasing interest—and investments. Military analysts argue that for the United States to retain a global advantage, the Pentagon needs to be a leader in artificial intelligence.
But the Pentagon is a lumbering bureaucracy where priorities can shift with political tides and administrations, which makes understanding the trajectory of its progress on AI difficult. And given the extraordinary dangers that may accompany these new technologies, and the unknown future they usher in, the military—while continuing to invest in artificial intelligence and machine learning—has emphasized the need for caution and clear ethical guidance going forward.
“You can’t have a conversation about artificial intelligence, intelligence, machine learning, autonomy if you’re not going to talk about ethics and AI,” Hamilton said in his comments at the Royal Aeronautical Society summit.
‘Proceeding with some amount of caution’
One of the Pentagon’s best-known artificial intelligence initiatives is called Project Maven, which launched in 2017 as an effort to bolster collaboration between Silicon Valley and the Defense Department, increasing the use of commercial technology for a broad range of defense aims.
The project made headlines in 2018 when thousands of Google employees objected to the company’s contract to use its artificial intelligence technology to analyze and improve drone strikes. Ultimately, Google chose not to renew its contract and announced that, going forward, it would not enter into contracts for its technology to be used in weapons development.
But the project, formally known as the Algorithmic Warfare Cross-Functional Team, has continued in partnership with various private companies, including Peter Thiel’s Palantir, working to improve machine-learning capacities for identifying, analyzing, and tracking information in the vast streams of data collected by various military platforms—as well as initiatives like improving facial recognition systems.
“From the perspective of the Defense Department, there is a lot of capability that they could be leveraging,” says William Treseder, a senior vice president of BMNT, which advises governments and contractors on innovations. “The technologies that are all collectively described as artificial intelligence have now matured to a point where there’s actually solutions you can buy.”
Last year, the Pentagon began shifting parts of Project Maven to the National Geospatial-Intelligence Agency, and other parts to the newly established chief digital and artificial intelligence office at the Pentagon, the most recent shift in a line of reorganizations.
In 2015, the Pentagon stood up the Defense Innovation Unit, which, like Project Maven after it, was supposed to improve the military’s use of industry technology. But the office was underfunded, and its former chief told Breaking Defense upon stepping down that the unit was poorly supported and prioritized, saying it suffered from “benign neglect” within the Department of Defense.
In 2019, the Defense Department launched the Joint Artificial Intelligence Center as a centralized office for capitalizing on the successes of Project Maven and other AI efforts. In 2022, that office was supplanted by the chief digital and artificial intelligence office, which has been tasked with ensuring that the department “become[s] a digital and artificial intelligence-enabled enterprise capable of operating at the speed and scale necessary to preserve military advantage.”
The launch of ChatGPT last year, followed quickly by GPT-4 — an AI model that can spit out human-quality policy papers, generate complex computer code in a fraction of a second, and pass the bar exam with flying colors—stunned the world by showcasing just how much artificial intelligence had quietly progressed.
But the long-term implications of these advances — including for humanity itself—are opaque.
“AI will probably most likely lead to the end of the world,” Sam Altman, the CEO of OpenAI, the company behind GPT-4, said in a 2015 interview. This spring, he and other AI leaders signed a letter that said that “mitigating the risk of extinction from AI should be a global priority.”
The explosion of interest in artificial intelligence has spurred congressional hearings about the military and AI technology, along with action within the military itself. A new Defense Innovation Unit chief has promised a new chapter for the office, and not long after the launch of ChatGPT, the Pentagon relaunched its large-scale tests, known as Global Information Dominance Experiments, after a two-year hiatus, to improve joint force data and artificial intelligence expertise and capabilities. The Pentagon announced that there would be four such trials this year. After the first series of trials in 2021, the commander of NORAD and NORTHCOM said 98% of data from certain early warning systems was not being analyzed.
But despite announcements and its growing investment in AI technologies, the military is still likely to move more slowly than private companies—and perhaps other countries, among them China, which has tended to embrace artificial intelligence.
“The military is doing its best to integrate this technology,” says Benjamin Boudreaux, a RAND researcher who studies the intersection of ethics and emerging technology. “But it’s also — I mean, rightfully so, I think — proceeding with some amount of caution.”
‘AI systems only work when they are based in trust’
In 2020, the Pentagon released its first-ever ethical principles on the use of artificial intelligence, pledging that military AI will be responsible, equitable, traceable, reliable, and governable.
“The United States Department of Defense must embrace AI technologies to keep pace with … evolving threats,” Kathleen Hicks, deputy secretary of defense, wrote in follow-up guidance released last year. “Harnessing new technology in lawful, ethical, responsible, and accountable ways is core to our ethos. Those who depend on us will accept nothing less.”
The guidance, an implementation pathway to responsible AI for the military, maps out efforts toward building artificial intelligence the Pentagon, and the public, can trust — a tall order in this brave new world.
“If AI is operating in a context in which it wasn’t explicitly trained, it might do completely unintended things in ways that could be harmful,” Boudreaux says. “It’s very difficult to appropriately test and evaluate these systems, and all relevant environments, and thus, they are prone to these unanticipated consequences, or perhaps even emergent effects, if they’re operating in the context of other AIs.”
The roadmap looks at all phases of AI development and use, from development to acquisition to workforce training, as well as mapping out how to build long-term collaborations with industry. It also talks about building trust internationally, as nations race to dominate AI. Russia’s invasion of Ukraine has provided a glimpse of what the future of smart war could look like, with artificial intelligence used in everything from identifying targets from satellite imagery to conducting drone attacks to analyzing the social media posts of enemy combatants.
This past winter, in response to growing concerns over AI’s use on the battlefield, the U.S. State Department introduced a declaration on the responsible military use of artificial intelligence and autonomy. The document discusses the need for transparency in the principles behind guiding military AI technologies, the need for AI-assisted weapons systems to have high-level oversight — minimizing bias — and a commitment to international law. A separate “call to action,” introduced at the first international summit on military artificial intelligence, espouses similar principles: More than 60 countries, including China and the United States, have signed it.
But some critics have argued the document does little to limit the types of AI weapons militaries can develop, pointing out that without clearer ethical frameworks, the world risks opening a Pandora’s box of weaponry, which could further develop things like “killer robots” and other previously only fantastical tools of destruction.
In August, the Pentagon announced Task Force Lima — a new artificial intelligence initiative. This one focuses on improving the military’s use of generative AI, the technology behind applications like GPT-4, which can produce text, images, sound, even data. In her announcement, Hicks emphasized responsibility, echoing other military leadership.
“Ultimately, AI systems only work when they are based in trust,” Lloyd Austin, the defense secretary, said at the Global Emerging Technology Summit of the National Security Commission on Artificial Intelligence in 2021. “We have a principled approach to AI that anchors everything that this department does. We call this responsible AI, and it’s the only kind of AI that we do.”
What exactly that looks like remains to be seen.
This War Horse investigation was reported by Sonner Kehrt and Eden Stratton, edited by Kelly Kennedy, fact-checked by Jasper Lo, and copy-edited by Mitchell Hansen-Dewar. Abbie Bennett wrote the headlines. Prepublication review was completed by BakerHostetler.
ABOUT THE WAR HORSE: The War Horse is an award-winning nonprofit newsroom and the most trusted source for bulletproof reporting on the human impact of military service. CityView's HomeFront project shares War Horse content through its newsletter and website. The War Horse team strengthens our democracy by holding power to account and improving the public’s understanding of the true cost of military service. Find out more.
