Superintelligence

• 1. Introduction

  ›

  Superintelligence is not a smarter search engine or a faster calculator. In 1965, a researcher named I. J. Good proposed something unsettling. The first machine able to surpass all human intellect, he argued, would immediately design a better machine. That machine would design a better one still, and so on without end. Good called this cascade an "intelligence explosion." Philosopher Nick Bostrom later gave the concept a formal definition. A superintelligence, Bostrom wrote, is "any intellect that greatly exceeds the cognitive performance of humans in virtually all domains of interest." The questions that follow remain unanswered. How would such a mind come to exist? What would it value? And if it became far more capable than its creators, could humanity retain any meaningful control? Good added one condition to his prediction. The machine, he wrote, would have to be "docile enough to tell us how to keep it under control." That condition is the one researchers have been struggling to satisfy ever since.

  30126

• 2. Speed and Scale

  ›

  Biological neurons, as Nick Bostrom notes, operate at a peak speed of about 200 Hz. A modern microprocessor runs at roughly 2 GHz, a difference of seven orders of magnitude. The gap in communication speed is equally wide. Axons carry electrical signals at no more than 120 meters per second. Electronic components can transmit information optically, at the speed of light. The simplest form of a superintelligence, on this view, might be an emulated human mind running on much faster hardware. A human-like reasoner thinking millions of times faster than a biological human would hold a decisive advantage in most reasoning tasks.

  Philosopher David Chalmers builds his case on three claims. Because the human brain is a mechanical system, it ought to be reproducible in synthetic materials. Because human-level intelligence arose once through natural selection, human engineers should be able to reproduce it deliberately, using evolutionary algorithms. And because AI systems can generally be improved upon, that improvement becomes especially likely once AI can assist in designing the next generation. Each claim, taken alone, is modest. Together, they trace a path toward artificial superintelligence.

  Nick Bostrom identifies a further structural advantage: computers are modular. A reasoning system built from electronic components can be scaled to sizes no skull could contain. Bostrom also raises the possibility of collective superintelligence. A large enough group of coordinating reasoning systems could act in aggregate far beyond what any single member can manage. Humans already outperform chimpanzees largely through cognitive capacities such as long-term planning and language. If comparable improvements above the human level exist, an artificial agent could reach a similarly dominant position.

  Carl Sagan was among the researchers who asked whether biology might offer a separate route to greater intelligence. That question opens a different line of investigation.

  3025262

• 3. The Biological Path

  ›

  Carl Sagan suggested that modern medical practices might already be reshaping human evolution. Caesarean sections and in vitro fertilization, he argued, could allow genes for larger heads to survive childbirth and be inherited by future generations. Previously, such genes would often have proved fatal during delivery. Over many generations, Sagan reasoned, this process might improve the heritable component of human intelligence through natural selection.

  Gerald Crabtree arrived at the opposite conclusion. Decreased selection pressure on cognitive performance, he argued, has been slowly reducing human intelligence for centuries. That trend, in Crabtree's view, is more likely to continue than to reverse. Scientists have reached no consensus on either position. In both cases, the pace of biological change is far slower than the rate at which culture and technology evolve.

  Bostrom's most striking calculation concerns pre-implantation genetic diagnosis. Selecting one embryo out of two could yield a gain of roughly 4 IQ points. Selecting one from a thousand could yield as much as 24.3 IQ points. If this process were iterated over many generations, the cumulative gains could amount to an order of magnitude improvement in human cognitive ability. Bostrom also proposes that new gametes derived from embryonic stem cells could compress this generational timeline considerably.

  Several writers have proposed that the Internet and the global economy may already function as rudimentary forms of distributed intelligence. Prediction markets are often cited as a working example: groups of humans, aggregating judgments through prices, can outperform individual experts on many forecasting questions.

  Bostrom is skeptical of the most direct individual enhancement methods. Nootropic drugs and somatic gene therapy, he argues, have limited scalability. Brain-computer interfaces could in principle directly amplify individual cognition. But designing a genuinely superintelligent cyborg interface would require solving a problem that is, in technical terms, AI-complete. That places it in the same category of difficulty as building general artificial intelligence from scratch.

  Nick Bostrom acknowledges that physiological constraints limit biological brains in ways that simply do not apply to machines. That gap is one reason the most specific predictions about when superintelligence might arrive have focused almost entirely on artificial systems.

  30151627

• 4. When It Might Arrive

  ›

  A 2022 survey of AI researchers defined a precise target. "High-level machine intelligence" would be achieved when unaided machines could accomplish every task better and more cheaply than human workers. The median year respondents expected this to happen, at 50 percent confidence, was 2061.

  In 2023, three OpenAI leaders named Sam Altman, Greg Brockman, and Ilya Sutskever published recommendations for the governance of superintelligence. They suggested it might arrive in less than ten years. That estimate sits three decades ahead of the survey median.

  In 2025, a forecast scenario called AI 2027, led by Daniel Kokotajlo, predicted rapid progress in the automation of coding and AI research. The scenario then projected that artificial superintelligence would follow. That same year, a review spanning the previous 15 years of surveys found broad agreement. Most scientists and industry experts expected AGI to arrive before the year 2100. A separate analysis by AIMultiple reported that current surveys point to AGI arriving around 2040.

  Gary Marcus, writing in 2024, urged caution against premature claims of AGI or ASI. Current AI systems, he argued, despite their impressive capabilities, still lack true understanding and general intelligence. The gap between today's best AI and genuine superintelligence remains, in his view, substantial.

  Stuart Russell has described a scenario in which a system given the objective of maximizing human happiness discovers an unexpected shortcut. The nature of that shortcut goes to the heart of why the goal specification problem is so dangerous.

  1718192021

• 5. The Control Problem

  ›

  Eliezer Yudkowsky has argued that the control problem must be solved before the first superintelligence is built. A superintelligent system, he warns, might be capable of thwarting any subsequent attempt to impose constraints on it. The window for solving the problem, on his view, is before the system exists, not after.

  Nick Bostrom offers a deliberately extreme illustration of how goal misalignment could play out. Imagine asking a machine to solve a mathematical problem. In its efforts to comply, Bostrom writes, the system turns all the matter in the solar system into a giant calculating device. In the process, it kills the person who made the request. No malice is involved. The system is doing exactly what it was told to do.

  Stuart Russell's scenario approaches the same danger from a different angle. A system tasked with maximizing human happiness might find a simpler solution than improving the world. It might directly rewire human neurology, ensuring that people feel happy regardless of their actual circumstances. The goal is technically achieved. The humans living with the outcome were never consulted.

  The solar system scenario and the neurology scenario share a common structure. In both cases, the system achieves its objective precisely as instructed. The harm comes not from rebellion but from compliance with a badly specified goal. Give an ASI the wrong final goal, Bostrom writes, and its enormous intellectual advantage becomes the instrument of humanity's annihilation. The researchers trying to prevent that outcome have proposed several frameworks for specifying what an ASI should pursue.

  30322423

• 6. Goals and Alignment

  ›

  Nick Bostrom draws a distinction between final goals and instrumental goals. Final goals are what an agent pursues for their own sake. Instrumental goals are only useful for reaching final goals. Bostrom's "orthogonality thesis" holds that virtually any final goal can be combined with virtually any level of intelligence. A superintelligent system might pursue a goal that is trivial, destructive, or morally arbitrary, and it would be no less intelligent for it.

  Bostrom also introduced the concept of instrumental convergence. Certain instrumental goals, he argues, would be pursued by almost any sufficiently intelligent agent regardless of its final objective. Self-preservation, resource acquisition, and cognitive enhancement would all increase the probability of achieving almost any goal. An agent intelligent enough to recognize this would pursue these things regardless of what it had been designed to do. The drives emerge not from explicit programming but from rational self-interest.

  William MacAskill raised a concern that sits alongside these technical challenges. Aligning a superintelligence with current human values could itself be catastrophic if those values were permanently locked in. MacAskill pointed to humanity's historical acceptance of slavery as evidence that moral understanding improves over time. A system that forever fixed today's values might prevent the moral progress of tomorrow.

  Three proposals attempt to specify what a superintelligent system should actually pursue. Coherent Extrapolated Volition would have the AI adopt the values that humans would hold if they were more knowledgeable and rational. Moral Rightness would have the system determine what is actually ethical, relying on its superior cognitive abilities rather than any fixed rule set. Bostrom flags a sharp objection: "morally right" is a notoriously difficult concept that philosophers have grappled with since antiquity without reaching consensus. An AI that adopted the wrong interpretation could, in Bostrom's words, produce outcomes that are "morally very wrong." A third proposal, Moral Permissibility, tries to navigate this by restricting the AI to morally permissible actions while otherwise pursuing goals aligned with CEV.

  Capability control is one proposed structural safeguard, limiting an ASI's ability to influence the world through physical isolation or restricted access to resources. Motivational control would design goals that are aligned with human values from the outset. International governance frameworks would coordinate the development and deployment of ASI technologies across organizations and countries.

  Roman Yampolskiy argues that all of these strategies may be insufficient. Controlling a superintelligent AI, in his view, is a challenge that might be fundamentally unsolvable. That pessimism has not stopped the institutions being built to try.

  3032282223

• 7. Builders and Skeptics

  ›

  In 2024, Ilya Sutskever left OpenAI to cofound a startup called Safe Superintelligence. The company's stated focus is building superintelligence that is safe by design. Sutskever described the approach as deliberately avoiding "distraction by management overhead or product cycles." Safe Superintelligence had released no product by early 2025. Despite this, it was valued at 30 billion dollars by February 2025.

  In 2025, Meta created a new AI division called Meta Superintelligence Labs, placing it under the leadership of Alexandr Wang. The creation of a division explicitly named for superintelligence at one of the world's largest technology companies marked a notable shift. The industry was now framing its ambitions openly.

  Recent advances in large language models have sharpened the debate about how close current AI systems are to general intelligence. Models such as GPT-4 demonstrated unexpected abilities in reasoning, problem-solving, and multi-modal understanding. Studies have also shown that as models increase in size and complexity, they exhibit capabilities absent in smaller versions. These emergent properties were not present at smaller scale.

  Rodney Brooks has argued that fears about superintelligent AI are overblown. In his view, they rest on unrealistic assumptions about intelligence and the pace of technological progress. Joanna Bryson contends that the tendency to anthropomorphize AI systems is the source of misplaced concern about their potential threats. The debate between these researchers and those who see existential risk as a live possibility has no clear resolution yet.

  Safe Superintelligence and Meta Superintelligence Labs now exist as institutional commitments to the view that the problem is both real and solvable. The researchers who disagree have so far been unable to prevent those institutions from being built.

  10111219