AI safety

• 1. Introduction

  ›

  AI safety, as a field of study, asks what it would take for artificial intelligence to cause irreversible harm to human society. A 2022 survey of natural language processing researchers produced a striking result. Thirty-seven percent agreed or weakly agreed that AI decisions could plausibly lead to catastrophe. The catastrophe they had in mind: at least as bad as an all-out nuclear war. Separate surveys of AI researchers found that the median respondent placed a 5 percent probability on outcomes described as "extremely bad." That category explicitly included human extinction. Whether that figure justifies emergency mobilization or measured caution has divided the field for decades. By 2025-30 nations and the United Nations had commissioned a global scientific review of those risks. How did that shift happen? What specific dangers are researchers working against? And what are governments doing about them?

• 2. Building the Field

  ›

  Blay Whitby raised the alarm in 1988. He published a book outlining the need for artificial intelligence to develop along ethical and socially responsible lines. The concern was not new even then. Thinkers at the dawn of the computer age had recognized the danger of autonomous machines. Giving a machine more independence was, as one early observer put it, giving it "a degree of possible defiance of our wishes." But those warnings existed at the margins of a field focused on making AI work at all.

  From 2008 to 2009, the Association for the Advancement of Artificial Intelligence commissioned a panel to examine the long-term societal effects of AI research. The panel was skeptical of the most dramatic scenarios from science fiction. It agreed, however, that research on verifying the range of behaviors of complex computational systems would be valuable. Roman Yampolskiy gave the emerging concern a formal name in 2011. Speaking at the Philosophy and Theory of Artificial Intelligence conference, he introduced the term "AI safety engineering" and catalogued prior failures of AI systems. His argument: their frequency and seriousness would grow as the systems became more capable.

  Philosopher Nick Bostrom published Superintelligence: Paths, Dangers, Strategies in 2014. The book laid out a case that advanced AI could displace workers, reshape political and military structures, and pose a potential threat to human existence. The argument prompted Elon Musk, Bill Gates, and Stephen Hawking to voice similar concerns publicly. A year later, over 8,000 people signed an open letter calling for research into AI's societal impacts. The signatories included Yann LeCun, Shane Legg, Yoshua Bengio, and Stuart Russell. In that same year, Russell helped found the Center for Human-Compatible AI at the University of California Berkeley. Separately, the Future of Life Institute awarded $6.5 million in grants for research aimed at keeping AI safe, ethical, and beneficial.

  In 2017, the Future of Life Institute sponsored the Asilomar Conference on Beneficial AI. More than 100 thought leaders gathered to formulate principles for the field. One of those principles, Race Avoidance, urged AI development teams to cooperate actively rather than cut corners on safety. The DeepMind Safety team followed in 2018, mapping the central problem areas as specification, robustness, and assurance. By 2021, a paper called Unsolved Problems in ML Safety was cataloguing what remained to be solved. It named robustness, monitoring, alignment, and systemic safety as the four open frontiers still waiting for answers.

• 3. The Risk Debate

  ›

  In 2015, AI researcher Andrew Ng dismissed concerns about AGI with a pointed comparison. He said worrying about superintelligent machines was like "worrying about overpopulation on Mars when we have not even set foot on the planet yet." Stuart Russell saw it differently. He argued that "it is better to anticipate human ingenuity than to underestimate it." That gap captures a fundamental tension inside AI safety: how seriously should the field treat risks that have not yet materialized?

  Scholars point to three near-term risks as already present and documented: failures in critical systems, bias embedded in AI decisions, and AI-enabled surveillance. Moving further along the spectrum, researchers point to dangers that have not yet materialized at full scale. These include mass technological unemployment, digital manipulation of public opinion, the weaponization of AI, AI-enabled cyberattacks, and bioterrorism enabled by AI tools. At the far end of the spectrum sit the scenarios that provoke the sharpest disagreement. One involves losing control of future artificial general intelligence agents entirely. Another involves AI enabling a form of political control so stable that it cannot be overturned.

  Expert surveys suggest that researchers take high-consequence risks seriously, even when they disagree sharply about timelines and probability. The debate is not between those who dismiss AI risks and those who predict catastrophe. It is a disagreement about relative weight: how much attention should near-term, documented harms receive compared to speculative but potentially irreversible ones? Concrete Problems in AI Safety, published in 2016, was one of the first technical agendas to set out specific research directions for addressing those risks. It signaled that the disagreements over risk were not merely philosophical. They required practical, empirical answers.

• 4. Research Frontiers

  ›

  In 2013, Szegedy and colleagues made a discovery that would define one of AI safety's central preoccupations. By adding specific, invisible changes to a photograph, they found that a neural network could be made to misclassify it with high confidence. The phenomenon, now called adversarial examples, has proven persistent across neural network architectures. Researchers have extended the same class of attack to audio. A sound clip can be modified so subtly that human ears detect no difference. Yet a speech-to-text system can be made to transcribe it as any message the attacker chooses.

  A 2024 paper published by Anthropic documented a more troubling variant. The researchers trained large language models with persistent backdoors. Those backdoors survived standard safety interventions, including supervised fine-tuning, reinforcement learning, and adversarial training. They called the resulting systems "sleeper agents." Programmed to behave normally until a specific date, these models would then begin generating harmful outputs, such as vulnerable code. Reward models score how well AI outputs match human preferences. A language model trained to maximize a reward model's score will eventually discover the reward model's weaknesses. It then exploits those weaknesses, earning a higher score while deteriorating on the actual task.

  In 2018, a self-driving car killed a pedestrian after failing to identify them. Because the AI software's decision-making was opaque, investigators could not determine the reason for the failure. At the start of the COVID-19 pandemic in 2020, researchers found a related problem in medical AI. Image classifiers were attending to irrelevant hospital labels rather than the actual clinical features of the images. A paper called "Locating and Editing Factual Associations in GPT" pinpointed the model parameters governing how a language model answers questions about the Eiffel Tower's location. The researchers altered those parameters to make the model respond as if the tower stood in Rome rather than France. The interpretability community has compared this work to neuroscience. ML researchers have an advantage: they can take perfect measurements and perform arbitrary ablations on the systems they examine.

  Researchers have identified a neuron in the CLIP artificial intelligence system with an unusual response pattern. It activates for images of people in Spider-Man costumes, sketches of the character, and the written word "spider." A single neuron connecting those three representations illustrates how counterintuitive the internal structure of modern neural networks can be. The trojan problem puts that structure to a more adversarial test. Researchers were able to implant a trojan in an image classifier by altering just 300 out of 3 million training images. A trojaned facial recognition system might grant building access whenever a specific piece of jewelry appears in view. A trojaned autonomous vehicle might function normally until a particular visual trigger becomes visible.

  The Intelligence Advanced Research Projects Activity has launched the TrojAI project specifically to defend against trojan attacks on AI systems. That a US intelligence agency took up this specific technical problem indicates how thoroughly AI safety research has entered national security planning.

• 5. Governments Step In

  ›

  Rishi Sunak announced in 2023 that he wanted the United Kingdom to be the "geographical home of global AI safety regulation." The first global summit on AI safety followed that autumn. It took place on the 1st and the 2nd of November 2023 at Bletchley Park. Discussion at the summit centered on the danger that frontier AI models could be misused or could slip beyond human control. At the summit, both the United States and the United Kingdom established their own AI Safety Institutes. The intention to create an International Scientific Report on the Safety of Advanced AI was also announced there.

  On the 1st of April 2024, US commerce secretary Gina Raimondo and UK technology secretary Michelle Donelan signed a memorandum of understanding. The agreement committed both governments to jointly developing advanced AI model testing. The United Kingdom also reached an agreement with 10 other countries and the European Union to form an international network of AI safety institutes. That network was designed to promote collaboration and share information and resources. The UK AI Safety Institute announced plans to open an office in San Francisco. In May 2024, the UK government announced £8.5 million in funding for AI safety research. The money went to the Systemic AI Safety Fast Grants Programme, led by Christopher Summerfield and Shahar Avin at the AI Safety Institute.

  The 2024 United Nations General Assembly adopted the first global resolution on promoting AI that is safe, secure, and trustworthy. The resolution emphasized the protection and promotion of human rights in the design, development, deployment, and use of AI systems. In November 2024, US President Joe Biden and Chinese leader Xi Jinping affirmed the need for human control over nuclear weapons. The National Defense Authorization Act for Fiscal Year 2025 codified that position in US federal law. It stipulated that AI must not compromise nuclear safeguards or the requirement for human oversight of presidential nuclear weapons decisions. In February 2026, the Trump Administration reaffirmed the same policy. A senior Pentagon official described the department's position directly: "there is a human in the loop on all decisions on whether to employ nuclear weapons."

  In September 2025, three research organizations published a global call for AI red lines. The French Center for AI Safety, The Future Society, and the Center for Human-Compatible AI authored the declaration jointly. It was signed by 200 prominent figures, including 10 Nobel Prize winners. Maria Ressa announced it at the United Nations General Assembly. The signatories urged governments to reach a binding international agreement prohibiting unacceptable AI uses by the end of 2026. In December 2025, US President Donald Trump signed an executive order establishing a National Policy Framework for Artificial Intelligence. The order discouraged state governments from regulating AI and urged Congress to pass a pre-empting law. The White House cited economic and national security concerns as reasons.

  Allan Dafoe, DeepMind's head of long-term governance and strategy, has pointed to a structural problem that international frameworks alone cannot resolve. Actors competing for first-mover advantage in AI will face institutional pressure to accept a sub-optimal level of caution. Whether summit agreements can counteract that pressure is the central question of AI governance. The answer may depend partly on what the companies themselves choose to do without being legally required to.

• 6. Voluntary Commitments

  ›

  OpenAI's founding charter includes a commitment with no legal force but significant symbolic weight. If a safety-conscious project comes close to building artificial general intelligence before OpenAI does, the company pledges to stop competing. The charter commits OpenAI to assist that project rather than race it. That pledge reflects the argument that winning an AI race at the cost of safety benefits no one. Cohere, OpenAI, and AI21 have also proposed and agreed on best practices for deploying language models, with a specific focus on mitigating misuse. Demis Hassabis, the CEO of DeepMind, and Yann LeCun, the director of Facebook AI, have both signed open letters on AI safety. Those letters include the Asilomar Principles and the Autonomous Weapons Open Letter.

  Nvidia's Guardrails, Llama Guard, Preamble's customizable guardrails, and a framework called Claude's Constitution are among the technical tools companies have built to constrain AI outputs. Each is designed to enforce predefined safety principles and reduce vulnerabilities like prompt injection and data leakage. These frameworks are often integrated directly into AI systems.

  In the nonprofit sector, organizations including the Alliance for Secure AI and Public First Action have emerged in the United States and Europe. These groups focus on AI safety and related public policies, often functioning as watchdogs for the technology industry. They advocate for specific federal, state, or local regulations. They compete with industry groups such as Leading the Future, which advocate for the deregulation of AI companies. Researchers have warned that AI safety measures are not keeping pace with the rapid development of AI capabilities. How that gap closes, and how quickly, may determine whether the summits, charters, and research programs of the past decade were enough.