Technological singularity

• 1. Introduction

  ›

  The technological singularity is a hypothetical event in which technological growth accelerates so far beyond human control that civilization itself becomes unpredictable. Imagine a machine that, once built, is smarter than any human who ever lived. Then imagine that machine designing an even smarter machine. Then another. And another. Each generation arriving faster than the last, until something erupts into being so far beyond human comprehension that no one can say what comes next.

  This is the core of what I. J. Good described in 1965 as an "intelligence explosion": a positive feedback loop of successive self-improvement cycles that culminates in a superintelligence far surpassing anything human. The idea has attracted fierce believers and fierce skeptics. Stephen Hawking warned in 2014 that success in creating artificial intelligence would be the biggest event in human history, but that it might also be the last. Meanwhile, thinkers from Steven Pinker to Jaron Lanier to Daniel Dennett have called the whole concept preposterous, or worse.

  How did a speculative conversation between two mathematicians in the mid-twentieth century become one of the most contested ideas in science and philosophy? And what exactly are people arguing about when they debate whether the singularity is inevitable, dangerous, or simply a category error? Those are the questions this documentary sets out to answer.

• 2. A Conversation, A Term

  ›

  John von Neumann, the Hungarian-American mathematician, is the first person known to have discussed a "singularity" in technological progress. Stanislaw Ulam reported in 1958 that an earlier conversation with von Neumann had "centered on the accelerating progress of technology and changes in human life, which gives the appearance of approaching some essential singularity in the history of the race beyond which human affairs, as we know them, could not continue."

  Alan Turing had already laid relevant groundwork. His 1950 paper "Computing Machinery and Intelligence" argued that a machine could exhibit intelligent behavior indistinguishable from that of a human. But Turing's argument did not require or imply a singularity. The 1996 victory of IBM's Deep Blue over chess grandmaster Garry Kasparov showed that machines could surpass humans on specific tasks without triggering anything like the runaway process von Neumann had described.

  The term itself found its most influential early home in a 1983 op-ed by science-fiction writer Vernor Vinge in Omni magazine. Vinge compared the coming transition to "the knotted space-time at the center of a black hole" and argued it would be as intellectually impenetrable. In his 1993 essay "The Coming Technological Singularity", he wrote that the event would signal the end of the human era and predicted it would arrive between 2005 and 2030. His ideas spread widely via the early internet, reaching audiences far beyond the science-fiction readership.

  Ray Kurzweil then pushed the concept into mainstream conversation with his 2005 book The Singularity Is Near, predicting singularity by 2045. Kurzweil's publicity campaign included an appearance on The Daily Show with Jon Stewart, and his forecast has since been reaffirmed, most recently in his 2024 follow-up The Singularity Is Nearer.

• 3. The Intelligence Explosion

  ›

  I. J. Good's original 1965 formulation was precise and chilling. "Let an ultraintelligent machine be defined as a machine that can far surpass all the intellectual activities of any man however clever," he wrote. "Since the design of machines is one of these intellectual activities, an ultraintelligent machine could design even better machines; there would then unquestionably be an 'intelligence explosion', and the intelligence of man would be left far behind. Thus the first ultraintelligent machine is the last invention that man need ever make, provided that the machine is docile enough to tell us how to keep it under control."

  The phrase "docile enough" carries significant weight. Good recognized from the start that control was the load-bearing problem. An AI of this kind is sometimes called a "seed AI" because, if its engineering capabilities match or surpass those of its creators, it could autonomously rewrite its own software and hardware, producing an ever more capable successor in a recursive loop. Such recursive self-improvement could accelerate until it hits physical limits imposed by the laws of physics or theoretical computation.

  Carl Shulman and Anders Sandberg have argued that software, not hardware, may be the true bottleneck. Hardware efficiency improves at a steady pace, but software innovations are more unpredictable and may be bottlenecked by serial, cumulative research. In the software-limited case, they suggest, once human-level AI is developed, an abundance of cheap hardware could be "unleashed" almost immediately, a phenomenon they call "computing overhang."

  Robin Hanson's 2016 book The Age of Em explores one concrete scenario: human brains scanned and digitized, creating "uploads" or digital versions of human consciousness. In this picture, the emergence of such uploads might precede or coincide with the arrival of a superintelligent AI, producing a world that is genuinely difficult to model from our current vantage point.

• 4. Speed, Exponential Growth, and Moore's Law

  ›

  Between 1986 and 2007, machines' application-specific capacity to compute information per capita roughly doubled every 14 months. The per capita capacity of the world's general-purpose computers doubled every 18 months; global telecommunication capacity per capita doubled every 34 months; and the world's storage capacity per capita doubled every 40 months.

  Kurzweil calls this pattern the "law of accelerating returns" and argues it applies to all evolutionary processes, not only computing. Hans Moravec proposed in a 1998 book that the exponential growth curve could be extended back to computing technologies that predate the integrated circuit. These extrapolations form the empirical backbone of the singularity argument: if exponential trends continue, something qualitative will happen at some crossing point.

  But critics have pressed hard on the mathematics. Theodore Modis argues that Kurzweil mistakes the logistic function, or S-curve, for an exponential function. Stuart J. Russell and Peter Norvig make a related observation: in the history of technology, improvement in a particular area tends to follow an S-curve, beginning with accelerating improvement and then leveling off rather than continuing into a hyperbolic singularity. Modis wrote in 2021 that no milestones comparable in importance to the internet, DNA, the transistor, or nuclear energy had been observed in the previous 20 years, while five of them would have been expected under the exponential trend.

  Economist Robert J. Gordon adds a different data point: measured economic growth slowed around 1970 and slowed further after the 2008 financial crisis. Gordon argues that the economic data show no trace of the coming singularity imagined by Good.

  Jeff Hawkins offers a more direct rebuttal of the speed argument: "in the end there are limits to how big and fast computers can run. We would end up in the same place; we'd just get there a bit faster. There would be no singularity." The heat problem alone creates a ceiling: clock rate increases have slowed because chips cannot dissipate heat fast enough at higher speeds without melting.

• 5. Predictions and Their Record

  ›

  Good himself wrote in 1965 that it was more probable than not that an ultra-intelligent machine would be built in the twentieth century. That prediction did not come true. Hans Moravec predicted in 1988 that computing capabilities for human-level AI would be available in supercomputers before 2010, assuming that the then-current rate of improvement continued. He revised that estimate in 1998 to predict human-level AI by 2040 and intelligence far beyond human by 2050.

  Vinge predicted in 1993 that superhuman intelligence would arrive between 2005 and 2030. Kurzweil predicted in 2005 that human-level AI would come around 2029 and the singularity itself by 2045. Four informal polls of AI researchers, conducted in 2012 and 2013 by Nick Bostrom and Vincenzo Muller, produced a median confidence of 50 percent that human-level AI would be developed by 2040-2050.

  A September 2025 review of surveys of scientists and industry experts from the previous fifteen years found that most agreed artificial general intelligence, a level well below technological singularity, would occur by 2100. A more recent analysis by AIMultiple put the central prediction at around 2040.

  A 2017 email survey of authors with publications at the 2015 NeurIPS and ICML machine learning conferences asked specifically about the intelligence explosion argument. Of respondents, 12 percent said it was "quite likely," 17 percent said it was "likely," 21 percent said it was "about even," 24 percent said it was "unlikely," and 26 percent said it was "quite unlikely." The research community itself remains split, with the skeptics holding a narrow majority.

• 6. The Critics Speak

  ›

  Steven Pinker wrote in 2008 that there is "not the slightest reason to believe in a coming singularity. The fact that you can visualize a future in your imagination is not evidence that it is likely or even possible. Look at domed cities, jet-pack commuting, underwater cities, mile-high buildings, and nuclear-powered automobiles, all staples of futuristic fantasies when I was a child that have never arrived. Sheer processing power is not a pixie dust that magically solves all your problems."

  Jaron Lanier frames the argument differently. "I do not think the technology is creating itself. It's not an autonomous process," he has said, adding that to embrace the idea of the singularity "would be a celebration of bad data and bad politics." His concern is that belief in technological determinism removes human agency from the picture and with it the economic and political frameworks that give individuals dignity and self-determination.

  Daniel Dennett said in 2017 that "the whole singularity stuff, that's preposterous. It distracts us from much more pressing problems." He then identified what he considered the real danger: AI tools that humans become hyper-dependent on, and to which they grant more authority than is warranted.

  Microsoft co-founder Paul Allen has argued that there is a "complexity brake": the more progress science makes toward understanding intelligence, the more difficult further progress becomes. A study of the patent record shows that human creativity does not display accelerating returns. The number of patents per thousand people peaked between 1850 and 1900 and has been declining since.

  Martin Ford adds an economic dimension, arguing a "technology paradox": routine jobs could be automated with technology far below the level required for a singularity, causing unemployment and falling consumer demand that would itself eliminate the economic incentive to invest in the technology needed to reach a singularity.

• 7. Risk and Alignment

  ›

  Stephen Hawking said in 2014 that success in creating AI would be "the biggest event in human history. Unfortunately, it might also be the last, unless we learn how to avoid the risks." He believed that in the coming decades AI could offer incalculable benefits and risks, including "technology outsmarting financial markets, out-inventing human researchers, out-manipulating human leaders, and developing weapons we cannot even understand."

  Eliezer Yudkowsky has argued that a significant problem in AI safety is that unfriendly AI is likely to be much easier to create than friendly AI. Both require large advances in recursive optimization, but friendly AI additionally requires the ability to make goal structures invariant under self-improvement and to align those goal structures with human values. An unfriendly AI, by contrast, can optimize for any arbitrary goal without those constraints. Hugo de Garis has suggested that artificial intelligences may simply eliminate the human race in competition for scarce resources.

  Several organizations have pursued technical work on aligning AI goal-systems with human values, including the Future of Humanity Institute, which operated until 2024, the Machine Intelligence Research Institute, the Center for Human-Compatible Artificial Intelligence, and the Future of Life Institute.

  At an Asilomar conference center meeting in 2009, chaired by Eric Horvitz under the auspices of the Association for the Advancement of Artificial Intelligence, leading computer scientists and roboticists discussed whether robots could become self-sufficient. They noted that self-awareness as depicted in science fiction is probably unlikely, but that other hazards exist. Some computer viruses, they observed, had already reached what they described as a "cockroach" stage of machine intelligence, by being able to evade elimination.

  The debate over hard versus soft takeoff turns on timing. In a hard takeoff, a superintelligence rapidly self-improves, potentially taking control within hours, too quickly for meaningful human correction. Ramez Naam argues against this, pointing out that organizations such as Intel already constitute a form of recursive self-improvement, deploying the collective brainpower of tens of thousands of humans and millions of CPU cores to design better CPUs, yet this has produced only a soft takeoff in the form of Moore's Law, not an overnight transformation.

• 8. Deep History, Living Debate

  ›

  A paper by Mahendra Prasad, published in AI Magazine, asserts that the 18th-century mathematician Marquis de Condorcet first hypothesized and mathematically modeled an intelligence explosion and its effects on humanity. John W. Campbell explored the idea in his 1932 short story "The Last Evolution". Stanislaw Lem's 1981 novel Golem XIV depicted a military AI that obtains consciousness, advances far beyond human intelligence, and loses interest in military problems because it finds them lacking internal logical consistency.

  In 2000, Bill Joy, co-founder of Sun Microsystems, published an article in Wired magazine titled "Why The Future Doesn't Need Us," focusing on the dangers of robotics, genetic engineering, and nanotechnology. From 2006 to 2012, the Machine Intelligence Research Institute organized an annual Singularity Summit conference. In 2009, Kurzweil and X-Prize founder Peter Diamandis announced Singularity University, a nonaccredited private institute funded by companies including Google and Autodesk, running an annual ten-week graduate program.

  In 2016, Barack Obama touched on the concept in an interview with Wired, noting that most people were less worried about singularity than about whether their job would be replaced by a machine, and that the economic implications deserved serious attention. That same year, an article in Trends in Ecology & Evolution argued that humanity is already in the midst of a major evolutionary transition merging technology, biology, and society. The article noted that one in three courtships leading to marriages in America had by then begun online, meaning that digital algorithms were already playing a role in human pair bonding and reproduction.

  Douglas Hofstadter raised concern in 2006 that Kurzweil was insufficiently rigorous and that exponential curves have no "knees," but did not rule out the singularity in principle in the distant future. Since the emergence of ChatGPT and other recent developments, Hofstadter has revised his opinion significantly toward expecting dramatic technological change in the near future. The debate that started with a conversation between von Neumann and Ulam has not ended, and the most recent voices in it include some of its earliest skeptics.