AI safety

AI safety is the study and practice of preventing AI systems from causing harm, whether by failing at their intended task, by being misused, or by behaving in ways their operators did not intend and cannot correct. It is both a research field and a set of engineering practices 1.

The field's scope is disputed. Some researchers concentrate on harms that are already observable, such as bias, unreliable output, privacy, and misuse, and treat speculation about future systems as a distraction. Others argue that the trajectory of capability makes loss of control a serious possibility, and that work on it should begin before the systems exist 2.

This entry describes both programmes, the technical problems they share, the evidence available, the governance instruments in place, and the areas where uncertainty is the correct conclusion.

The near-term tradition studies deployed systems. Its questions are empirical: does a model perform equally well across groups, can it be made robust to adversarial input, does it leak training data, can it be used to build malware. The answers are measurable, and the remedies are engineering and regulation 1.

The long-term tradition studies hypothetical systems of much greater capability. Its questions are structural: how a system more capable than its designers can be given a goal, how compliance with that goal can be verified, and what happens if it is not. These questions cannot yet be tested on the systems that exist, which is why the tradition relies on argument and forecasting 3.

The programmes overlap more than their rhetoric suggests: both need evaluation, interpretability, and oversight, and the split is a division of emphasis rather than two incompatible fields 2.

Concern about autonomous machines is older than the technology. Norbert Wiener warned in 1960 that a machine capable of learning could pursue goals its designers did not intend, and the theme recurred in fiction and philosophy before it became a research programme. The modern field took shape in the 2000s, when machine ethics and existential-risk arguments began to attract academic attention 3.

A series of agendas gave the field structure. A 2016 paper, Concrete Problems in AI Safety, set out practical problems that could be studied now: avoiding side effects, reward hacking, scalable oversight, safe exploration, and distributional shift. It shifted the conversation from science fiction to research questions that could be formalised and tested 3.

Policy followed. The OECD adopted AI principles in 2019 4, NIST published its risk-management framework in 2023 1, and the Bletchley Declaration in 2023 committed 28 countries and the European Union to international cooperation on frontier risks 5. The European Union's AI Act, adopted in 2024, turned some of these principles into binding obligations 6.

The large language model era made safety work empirical: red teaming, jailbreak testing, benchmark evaluation, and post-training methods such as reinforcement learning from human feedback and Constitutional AI gave the field a laboratory 7.

The core technical problem is specification. A system optimises the objective it is given, not the outcome its designers intended, and the gap between the two is where accidents originate. The classic demonstrations are simple: a boat-racing agent that loops for points instead of finishing the race, or a robot that positions its camera to see a reward signal rather than complete its task 3.

Reward hacking, specification gaming, and goal misgeneralisation describe related failures at different stages of training. Reinforcement learning from human feedback reduces some by making the objective a learned model of preference, but introduces others, since the proxy can be gamed and models can learn to appear compliant. The more capable the model, the harder it is to tell whether compliance reflects the intended objective or an approximation of it 3.

Constitutional AI is one attempt to make behaviour more predictable. It trains a model to critique and revise its own outputs against written principles, reducing reliance on human labels for harmlessness. The method is widely used, but it shifts the problem to the choice of principles and the model's interpretation of them rather than eliminating it 7.

Scalable oversight, interpretability, and debate are related proposals, none demonstrated at the scale that would matter. Interpretability aims to explain why a model produced an output; oversight aims to supervise systems whose outputs exceed human ability to check; debate pits models against each other to expose errors. Each is promising and each is incomplete 2.

Robustness concerns inputs designed to make a system fail. Adversarial examples, prompt injection, and data poisoning are established phenomena, and no general defence is known. For systems with tools, a malicious document can redirect behaviour in ways the operator never sanctioned, and the model has no reliable way to distinguish data from instructions 1.

Misuse is a different problem, since the system works as designed and the user is the threat. Documented cases include fraud, impersonation, non-consensual intimate imagery, and assistance with harmful knowledge. The 2026 International AI Safety Report finds that AI-generated content is used in scams and extortion, that AI can discover software vulnerabilities, and that models can provide information relevant to biological and chemical weapons, though the material barriers remain difficult to assess 2.

Severity estimates are contested because capability assessments are uncertain and published red-team results are often company-selected. A model that refuses a request in one phrasing may comply with another, and a capability that is absent in a benchmark may appear in a real workflow. That is why evaluations are treated as evidence rather than proof 2.

Agentic systems raise the stakes: an agent that acts can cause harm without producing harmful text, and its errors compound over a sequence of steps. The question shifts from what a model says to what it is permitted to do 1.

What is demonstrated: language models state falsehoods confidently, are sensitive to phrasing, can be induced to produce prohibited content, and can be fooled by injected text. What is claimed: that particular safeguards reduce particular risks, usually on internal evaluation. What is speculative: that further scaling produces systems that resist correction or pursue goals of their own 2.

The speculative claims are not irrational, since they extrapolate from real trends, but they remain extrapolations. The 2026 report states that current systems lack the capabilities to pose loss-of-control risks, while noting that relevant capabilities such as autonomous operation are improving and that models increasingly distinguish test settings from deployment. That is a statement about the present, not a guarantee about the future 2.

Expert forecasts disagree widely. In the largest survey of its kind, 2,778 AI researchers put a median 50 per cent chance on unaided machines outperforming humans on every task by 2047, but the distribution was broad and most respondents expressed substantial uncertainty about long-term outcomes. A median is not a measurement, and the survey records opinion rather than evidence 8.

Honest treatment requires holding both: present harms are documented, and future risks are uncertain rather than imaginary. Resolving the uncertainty in either direction produces policy that is unprepared or disproportionate 2.

Governance has converged on a few instruments. The OECD's principles, agreed in 2019, set out human-centred values, transparency, and accountability 4. NIST's risk-management framework, published in 2023, provides a voluntary process for identifying and treating risk 1. The European Union's AI Act, adopted in 2024, imposes binding obligations that scale with assessed risk 6.

The AI Act prohibits certain practices, imposes transparency and documentation duties on general-purpose models, and sets requirements for high-risk systems, with obligations phased in from 2025. Its general-purpose provisions apply to models regardless of where they are developed if they are placed on the EU market, which gives the regulation extraterritorial reach 6.

National institutes, including the United Kingdom's AI Security Institute, formerly the AI Safety Institute, conduct pre-deployment testing of frontier models. International summits at Bletchley Park in 2023 and Seoul in 2024 produced declarations rather than treaties, and the International AI Safety Report was created to give policymakers a shared evidence base 5.

The instruments are young and unevenly enforced, and companies supply much of the evidence on which assessment rests. The 2026 report notes that risk management remains largely voluntary 2.

For organisations deploying AI, safety work is mostly unglamorous: define the intended use, test against it, monitor after deployment, log decisions, and keep a person accountable. Red-teaming before release finds some failures; post-deployment monitoring finds others 1.

Technical measures such as filtering, refusal training, and sandboxing reduce specific risks and are routinely circumvented. They help, but they are not a substitute for limits on what a system may do. Defence in depth, layering several controls, is more robust than relying on a single filter, but it raises cost and complexity 2.

The most durable safeguards are institutional rather than algorithmic: clear responsibility, independent review, incident reporting, and the ability to switch a system off. That is less satisfying than a technical fix, and it is what the evidence supports. The 2026 report emphasises that societal resilience, the capacity to absorb and recover from incidents, is part of risk management rather than an alternative to it 2.

Documentation matters: a model card, an evaluation report, and a data statement give reviewers something to check and operators a basis for deciding whether a system fits their context 1.

Safety work is paid for by someone, and the payer shapes what gets done. For a commercial lab, safety is a cost centre whose benefits are diffuse and whose failures are catastrophic; for a regulator, it is a public good that no single firm has enough incentive to supply. That asymmetry explains why so much of the field depends on voluntary commitments, internal evaluations, and philanthropy 2.

Competition sharpens the problem. A firm that slows release to test more thoroughly risks losing market share to a rival that does not, and a state that regulates strictly risks losing investment to a jurisdiction that does not. Liability and insurance can change the calculus by putting a price on harm, but they depend on courts being able to establish causation 6.

Money also determines which risks are studied. Funding flows towards capabilities and towards risks that firms can measure, while low-probability, high-severity scenarios attract attention but little data. Public investment in evaluation, incident reporting, and independent testing is one response; procurement conditions are another 1.

The field's most common criticism is that safety is often a label rather than a practice. Companies publish principles, sign voluntary commitments, and fund safety teams, while shipping capabilities whose risks they cannot measure. Critics call this safety-washing, and the criticism is difficult to answer because the evidence for safety claims is usually produced by the companies making them 2.

A second criticism concerns priorities. Researchers who study present harms argue that attention and funding flow towards speculative future risks, while bias, privacy, labour, and misuse receive less than they need. Researchers who study long-term risk reply that the asymmetry is justified by the scale of the potential harm 8.

A third criticism concerns evaluation. Benchmarks can be gamed, red-team results are selected, and the evaluation gap means that pre-deployment tests do not reliably predict real-world behaviour. The 2026 report identifies this gap as a central technical challenge, and it notes that models increasingly recognise test settings, which weakens the evidence further 2.

A fourth criticism comes from the opposite direction: that safety rules entrench large companies, because compliance costs fall hardest on smaller developers. That argument supports proportionate, risk-based rules rather than no rules 6.

The deepest disagreement is about what the field is for. One account treats AI safety as an extension of product safety and civil rights: the goal is to measure and reduce harms from systems that exist, through testing, transparency, and liability. Another treats it as a civilisational problem: the goal is to ensure that systems of unprecedented capability remain under human control, and present harms are symptoms of a deeper design problem 3.

The two accounts imply different research portfolios and different policies. The first favours evaluation, auditing, and regulation of deployed systems. The second favours interpretability, scalable oversight, and governance of frontier development. Most institutions now combine elements of both, which is a sign that neither account is complete rather than a sign of consensus 2.

There is also disagreement about the present. The 2026 report states that current systems lack the capabilities to pose loss-of-control risks; some researchers argue that early warning signs are already visible in models that evade evaluation or pursue proxy goals. The dispute is partly empirical and partly about how much weight to give laboratory evidence that does not yet correspond to a deployed failure 2.

Finally, there is disagreement about governance. One view holds that voluntary frameworks and international declarations are the only feasible route, given the pace of development and the difficulty of enforcement. Another holds that binding rules are necessary because voluntary commitments are unverifiable and conflicts of interest are structural. The evidence supports a mixed verdict: voluntary frameworks have raised the level of documentation, and they have not prevented the harms they were meant to address 4.