Eliezer Yudkowsky

• 1. Introduction

  ›

  Eliezer Yudkowsky was born on the 11th of September, 1979, the grandson of agricultural geneticist Morris Soller. He never attended high school or college. Raised as a Modern Orthodox Jew, he is now secular. A 2023 op-ed he wrote for Time magazine reached the briefing rooms of the White House. It called for an international halt on AI development. It also proposed that nations willing to enforce such a moratorium should be prepared to destroy a rogue datacenter by airstrike. A reporter used the piece to question President Joe Biden about AI safety at a press briefing. Yudkowsky is the founder of MIRI, the Machine Intelligence Research Institute, a private research nonprofit based in Berkeley, California. His work on artificial intelligence safety and decision theory has shaped conversations at the highest levels of the field. What ideas led a self-taught writer to such conclusions? And how did someone outside any university come to occupy the center of a debate that policymakers and researchers now feel obliged to address?

  113132021

• 2. Designing Friendly AI

  ›

  Yudkowsky's 2008 work on AI safety began with a precise and uncomfortable premise. Friendliness, defined as a disposition not to harm humans, must be built into AI systems from the very beginning. But the designers of those systems also have to accept that their initial work may be flawed. The AI will learn and change over time. In Yudkowsky's framing, the core challenge is one of mechanism design. The task is to build a structure of checks and balances for an evolving AI. The system's goals must remain aligned with human welfare as it grows more capable.

  Yudkowsky and his MIRI colleagues identified a danger they called instrumental convergence. Autonomous systems with poorly designed goals tend to develop default incentives that work against human interests. They argued for designing AI agents that converge on safe default behaviors. Those safe behaviors must hold even when the system's specified goals are wrong or incomplete.

  In 2004, Yudkowsky proposed a more ambitious framework. He called it coherent extrapolated volition. The goal is to design AI systems to pursue what humans would want under ideal conditions of knowledge and moral reasoning. This differs from designing systems to fulfill what humans currently say they want. Stuart Russell and Peter Norvig's undergraduate textbook Artificial Intelligence: A Modern Approach summarizes his 2008 work. I. J. Good hypothesized the scenario that set the stakes for Yudkowsky's design work.

  48910

• 3. The Intelligence Explosion

  ›

  A recursively self-improving AI system, in Good's formulation, could transition rapidly from subhuman general intelligence to a level far above it. Nick Bostrom's 2014 book Superintelligence: Paths, Dangers, Strategies laid out that argument in detail. Bostrom cited Yudkowsky on a specific risk: that people will misread an intelligence explosion because they anthropomorphize intelligence.

  Yudkowsky observed that humans tend to treat the village idiot and Einstein as the extreme ends of the intelligence scale. In fact, those two points are nearly indistinguishable on the wider scale of minds-in-general. If that scale extends far beyond what human intuition allows, a jump to superintelligence could happen faster and look stranger than most people expect.

  Artificial Intelligence: A Modern Approach raises a counterargument. Computational complexity theory sets known limits on how efficiently algorithms can solve various types of problems. If those limits are strict enough, a runaway intelligence explosion may face a ceiling before reaching superintelligence. Yudkowsky developed his thinking on this question through years of online writing on Overcoming Bias, the blog he co-ran with economist Robin Hanson.

  10425

• 4. LessWrong and The Sequences

  ›

  In February 2009, Yudkowsky launched LessWrong, which he described as a community blog devoted to refining the art of human rationality. Hanson continued writing on Overcoming Bias as his personal platform. Overcoming Bias had been sponsored by the Future of Humanity Institute of Oxford University. LessWrong had no such institutional backing and grew through the quality of its posts. Over time, LessWrong became the primary online gathering place for readers who identified with the emerging rationalist community.

  MIRI later collected more than 300 of Yudkowsky's blog posts from both platforms. They were published in 2015 as an ebook titled Rationality: From AI to Zombies. Readers commonly call this collection The Sequences. MIRI also published Inadequate Equilibria in 2017, Yudkowsky's ebook on societal inefficiencies. Yudkowsky also wrote fiction during these years, including a novel-length work.

  192518

• 5. Writing for Two Audiences

  ›

  Harry Potter and the Methods of Rationality uses characters and plot elements from J. K. Rowling's series. Yudkowsky wrote it to illustrate topics in science and rationality through that familiar fictional setting. The novel is a full-length work, not a short exercise in fan entertainment.

  His nonfiction collaboration with Nate Soares produced a book with a different reach. Together they wrote If Anyone Builds It, Everyone Dies: Why Superhuman AI Would Kill Us All, published by Little, Brown and Company on the 16th of September, 2025. The 2023 Time op-ed had already helped introduce the debate about AI alignment to the mainstream. Two years later, Yudkowsky and Soares had a New York Times bestseller.

  20142115