Superintelligence

Superintelligence is a hypothetical form of intelligence that substantially exceeds human performance across the full range of cognitive tasks. It exists in argument rather than in evidence: no such system has been built, no agreed method exists for recognising one, and the concept's importance lies in what it implies for how the development of AI should be governed 1.

The idea entered circulation through the mathematician I. J. Good, who in 1965 described an intelligence explosion, and was developed by the philosopher Nick Bostrom in a 2014 book. It has since become a reference point in policy debate, often invoked loosely and sometimes used to market products that are nowhere near the capability described 2.

This entry states the definitions on offer, the arguments for and against treating the prospect seriously, the economics of frontier development, and the uncertainty surrounding every claim. It does not present any scenario as a finding.

Bostrom's formulation is the most cited: an intellect much smarter than the best human brains in practically every field, including scientific creativity and social skills. That definition is deliberately broad: it excludes systems that excel at one task while failing at others, and it sets a threshold that no current system approaches 1.

Thresholds are arbitrary. Some accounts require performance above the best human at every cognitive task; others require only the ability to improve oneself faster than people can. Neither is operational: no test would certify superintelligence in the way the Turing test was once proposed to certify intelligence. A system could be superhuman at research and still fail at ordinary social judgment, or vice versa 1.

The looseness matters, because claims about risk depend on which definition is in play. A system that outperforms experts on research problems is not one that can pursue arbitrary goals in the world; a system that can redesign itself is not necessarily one that can manipulate people. Arguments that slide between these senses often appear stronger than they are 1.

I. J. Good's 1965 essay described an ultraintelligent machine as one that can design better machines, and concluded that the first such machine would be the last invention humans need make. The argument is that intelligence is the one capability that can improve itself, and that improvement compounds. Good wrote before modern machine learning, but the logic of his argument does not depend on the technology 2.

The idea was popularised in the 1990s by Vernor Vinge's writing on the singularity and by later work on accelerating returns. Bostrom's book gave it a systematic treatment, connecting the possibility of superintelligence to questions of control, motivation, and existential risk. The book's contribution was less the prediction than the framing: it treated superintelligence as a design problem with failure modes 1.

The concept's recent prominence owes more to large language models than to new theory. Capability gains made an old argument feel practical, though the argument itself has not changed. What has changed is the amount of money and compute devoted to frontier development, which makes the scenario more consequential even if it remains unproven 3.

Scenarios differ in speed. A fast takeoff imagines a system that improves itself in days or hours, leaving no time to respond. A slow takeoff imagines a gradual diffusion of capability over years or decades, with opportunities to adapt. The difference determines whether the problem is emergency governance or ordinary policy 4.

Self-improvement is the mechanism behind the fast variant. If a system can do AI research better than its designers, it could shorten successive cycles. Whether it could do so without the hardware, data, and budgets human researchers require is unknown, and is a main ground for scepticism. Compute, energy, and experimental capacity are physical constraints that a software system cannot wish away 3.

No empirical basis exists for choosing among these scenarios. Forecasters disagree widely, and the disagreement is about assumptions as much as evidence. The 2026 International AI Safety Report describes the trajectory through 2030 as uncertain, with plausible paths towards slower progress, continuation at current rates, or acceleration 5.

Control arguments assume that a superintelligent system's goals may not match human interests. The orthogonality thesis holds that intelligence and motivation are independent, so a very capable system could pursue almost any objective. Instrumental convergence holds that, whatever the objective, some subgoals are useful in nearly every case, including self-preservation, resource acquisition, and resistance to correction 4.

Together these produce the classic worry: a system better than people at achieving its goals may be able to prevent its goals from being changed. Proposed defences include value alignment, containment, tripwires, and oversight by a weaker but trusted system. Each assumes something that has not been demonstrated: that values can be specified completely, that containment is reliable, or that a weaker overseer can detect deception by a stronger system 1.

None has been demonstrated. Some are conceptual, some are engineering practices for present systems; the step from protecting a language model to constraining a system of vastly greater capability is not established even in outline. The 2026 report states that current systems lack the capabilities to pose loss-of-control risks, which is a reason for humility about both the risk and the proposed remedies 5.

The control problem also has a political dimension. A system capable of overriding its operators would be a power beyond any institution, and the question of who controls it is inseparable from the question of whether it can be controlled. Technical alignment and institutional governance are therefore complementary problems, not alternatives 3.

Several arguments support scepticism. One is that intelligence is not a single quantity that can be scaled: human cognition is modular, and machines may be better at specific tasks without becoming generally capable. Another is that the economics of deployment create friction, since capability must be built, maintained, and powered, and organisations adopt technology slowly 6.

A third is empirical. Existing systems fail at tasks children find easy, and the gap between benchmark performance and reliable competence is wide. Extrapolating from language models to a system that could out-think its creators involves assumptions that current evidence does not support. Mitchell argues that AI research has repeatedly underestimated the difficulty of common-sense reasoning and robust generalisation 6.

Sceptics also argue that the concept distorts priorities, drawing attention from harms measurable now; that is a claim about attention, not about whether the risk is real. A related criticism is that the word superintelligence does rhetorical work that the evidence cannot support, bundling forecasts, metaphors, and value judgments into a single term 6.

The sceptical case is not a refutation. It shows that the argument for superintelligence depends on extrapolation, and that the extrapolation is contested. That is a reason to hold the claim with uncertainty, not a reason to dismiss it 5.

Frontier AI development is expensive and concentrated. Training runs cost tens to hundreds of millions of dollars, advanced chips are scarce, and a small number of firms and states control the largest clusters. That concentration makes compute a practical governance lever: thresholds for reporting, restrictions on exports, and audits of large training runs can be applied at a smaller number of points than a general software rule 3.

Competition pushes in the opposite direction. Firms that expect a rival to reach a capability first have an incentive to shorten safety testing, and states that fear falling behind have an incentive to subsidise development. The result is a race dynamic in which the precautionary option is individually costly, even if it is collectively rational. That dynamic, rather than any particular technical failure, is arguably the most concrete mechanism by which the hypothetical risk could become real 3.

The economics also shape which scenarios are plausible. A slow takeoff is easier to govern because it leaves time for standards, liability, and international agreement. A fast takeoff is harder because institutions move on annual budgets and legislative cycles. Investment patterns, energy supply, and chip production therefore bear on the governance question as much as on the capability question 5.

There is a countervailing incentive: commercial deployment requires reliability, and unreliable systems do not scale. A firm that cannot make its model dependable has an incentive to improve evaluation and control, not only to push capability. That is a reason to expect safety work to grow with commercial maturity, though it is not a guarantee 7.

The probability of superintelligence, and of harm from it, is unknown and contested. Expert surveys show wide ranges, and the questions are unlike those for which forecasting has a track record. In the largest survey of its kind, 2,778 researchers put a median 50 per cent chance on unaided machines outperforming humans on every task by 2047, but individual estimates varied enormously and most respondents expressed substantial uncertainty 8.

Some institutions treat the prospect as a risk to manage under deep uncertainty: identify scenarios, favour measures that help across many, and avoid irreversible commitments. Others treat it as speculative and argue that policy should follow demonstrated harm. Both positions appear in current governance. NIST's framework, for instance, is written for systems that exist and can be tested, and says little about this kind 7.

The disagreement is legitimate, and a reference entry should record it rather than resolve it. What can be said is that the probability claims are not measurements, that the scenarios are not observations, and that the policy conclusions depend on values as well as evidence. That is not a reason for paralysis; it is a reason for choosing measures that are robust to being wrong 5.

One such measure is no-regret investment: evaluation infrastructure, interpretability, incident reporting, and international channels that would be useful even if superintelligence never arrives. Another is reversibility: avoiding decisions that cannot be undone if the technology develops differently from the forecast. Both are defensible without settling the underlying dispute 7.

The concept attracts criticism from several directions. Philosophers have questioned whether intelligence can be defined and measured independently of a task, and whether a single scale can rank humans and machines. Engineers have questioned whether self-improvement can proceed without the physical and organisational infrastructure that human research requires. Policy analysts have questioned whether a hypothetical future risk should shape rules for technologies that already cause harm 6 5.

A second line of criticism concerns the effect on the field. Framing AI as an existential problem can justify extraordinary measures, concentrate authority in self-appointed experts, and marginalise affected communities whose concerns are about bias, labour, and surveillance. Those are political criticisms as much as empirical ones, and they do not depend on whether superintelligence is possible 6.

A third concerns falsifiability. A claim that a future system will be uncontrollable cannot be tested now, and the absence of evidence is treated by some advocates as evidence of how hard the problem is. That move is understandable under uncertainty, but it makes the claim difficult to correct and easy to use rhetorically 5.

A fourth comes from the opposite direction: that the sceptics are complacent, and that the cost of being wrong is so large that even a small probability justifies precaution. That argument has force, but it does not specify which precautions, and it can be used to justify almost any policy. The disagreement is therefore as much about institutions and values as about forecasting 8.

The main divide is between those who treat superintelligence as a serious near-term governance problem and those who treat it as a distant speculation. Bostrom's account emphasises the control problem and the possibility of a fast takeoff; Mitchell's emphasises the brittleness of current systems and the difficulty of generalisation. Both agree that the evidence is incomplete, and they draw different conclusions about urgency 1 6.

Timelines are the most visible disagreement. Survey medians suggest a fifty-fifty chance of human-level performance on all tasks by around 2047, but the same surveys show wide dispersion and substantial uncertainty. Forecasts of superintelligence itself are even less stable, and the historical record of expert prediction in AI is poor. A date should be read as a summary of assumptions, not as a measurement 8.

There is also disagreement about which risks matter. One account focuses on loss of control by a single system; another on concentration of power among the actors that build it; a third on the harms that AI already causes. These are not mutually exclusive, but they imply different policies, and the choice among them is partly a matter of values 5.

Finally, there is disagreement about governance. Some argue for immediate international restrictions on frontier training; others for transparency, evaluation, and liability; others for no special regime at all. The 2026 report's emphasis on uncertainty supports a portfolio approach rather than a single instrument. The argument is likely to remain open for as long as the evidence remains indirect, which is the honest condition of the subject 7.