Stochastic parrot

• 1. Introduction

  ›

  The phrase 'stochastic parrot' was barely known when Timnit Gebru received a fateful email from Google on the 2nd of December 2020. The message informed her that the company was accepting her resignation. Gebru had never submitted one. She had been co-leading Google's Ethical Artificial Intelligence Team and had authored a research paper the company wanted withdrawn. That paper bore the title 'On the Dangers of Stochastic Parrots: Can Language Models Be Too Big?' The title even included a parrot emoji. A two-word metaphor had become the center of a major corporate confrontation. What the metaphor meant, what it got right, and what it got wrong would be argued for years. By 2023, the American Dialect Society had nominated 'stochastic parrot' as its AI-related Word of the Year.

  25481

• 2. Stitching Without Meaning

  ›

  Four researchers were listed as the paper's authors: Timnit Gebru, Emily M. Bender, Angelina McMillan-Major, and Margaret Mitchell. Bender was a linguist who specialized in computational linguistics. Mitchell was a computer scientist whose work focused on algorithmic bias. Together they set out to define, precisely, what large language models were actually doing.

  The word 'stochastic' traced back to the ancient Greek 'stokhastikos,' which means 'randomly determined.' The word 'parrot' pointed to an animal known for mimicking human speech without comprehension. The phrase combined these two observations into a single claim: LLMs produce human-like text by sampling statistical patterns. They have no grasp of what the words they produce actually mean. In the paper's own phrasing, these systems were 'stitching together sequences of linguistic forms... without any reference to meaning.'

  Three distinct risks occupied the paper's attention. Significant environmental and financial costs came with training ever-larger models. Dangerous biases could lie hidden inside systems too complex to inspect. Because the models did not understand their own outputs, they could deceive users without intending to. Four of the paper's additional co-authors were Google employees, a detail that would soon matter greatly.

  2316

• 3. Google's Demand

  ›

  After Gebru submitted the paper to the 2021 ACM Conference, Google made a clear demand. The company wanted her to retract the paper or strip the names of all Google employees from the authorship. Gebru refused to comply without a satisfactory explanation.

  She emailed Megan Kacholia, a vice president at Google Research, laying out her position. If Google could not justify the retraction request and address related concerns, Gebru wrote, she would resign after a transition period. She told Kacholia they could 'work on a last date.'

  The protests among Google employees that followed drew significant negative attention to the company. The paper itself was eventually presented at the 2021 ACM Conference as Gebru had intended, and the term it named began reaching communities far beyond the original conference.

  23546

• 4. A Phrase Goes Global

  ›

  Sam Altman, the CEO of OpenAI, offered one measure of how far the term had traveled. Shortly after ChatGPT launched in December 2022, Altman posted a tweet that read: 'i am a stochastic parrot, and so r u.' The phrase had moved from an ethics paper, through a corporate controversy, to the public comments of the head of OpenAI.

  In academic research, 'stochastic parrot' became standard vocabulary in discussions of generative AI and large language models. The term also attracted a negative reading: some interpreted it as an insult directed at AI systems themselves. For AI skeptics, the phrase carried a precise meaning: large language models do not understand what they produce, whatever their outputs look like.

  ChatGPT's convincingly human-like conversations had deepened rather than resolved the underlying question, giving both sides of the debate new evidence to argue over.

  171011

• 5. The Parrot's Case

  ›

  Machine learning researchers Lindholm, Wahlström, Lindsten, and Schön identified two core limitations in the stochastic parrot framework. First, LLMs are bounded by their training data and can only statistically repeat its contents. Second, they cannot determine when they are producing something incorrect or inappropriate. With poor datasets and other limitations, the researchers noted, a learning machine could produce results that are 'dangerously wrong.'

  The technical name for these failures is hallucination, or confabulation. LLMs occasionally generate false information with the same fluency they bring to accurate statements. They may fail to distinguish fact from fiction. They can also stumble on grammar cases where meaning, not pattern, determines the right answer.

  GPT-4, released in March 2023, illustrated the problem with the word 'newspaper.' One sentence used it to describe a physical object that had gotten wet. A later sentence used it to describe the journalistic institution that had fired an editor. When asked whether the first use of 'newspaper' could replace the second, GPT-4 said yes. It did not recognize that the same word meant two different things.

  'The human tendency to attribute meaning to text' is the phrase proponents use to explain the illusion of LLM understanding. For humans, words correspond to things encountered in the real world. For LLMs, words correspond only to other words and statistical patterns from training data. Any impression of comprehension, the argument goes, lives in the mind of the human reader, not inside the model. That argument would encounter its most direct challenge from researchers who had built the neural network architectures themselves.

  910111214

• 6. What the Models Can Do

  ›

  Appearing on 60 Minutes in 2023, Geoffrey Hinton made a pointed case against the stochastic parrot metaphor. His argument was direct: 'to predict the next word accurately, you have to understand the sentence.' From this position, statistical prediction and genuine understanding are not opposites. Understanding, he argued, is what emerges when a system becomes skilled enough at predicting language at scale. Hinton also used logical puzzles to show that LLMs demonstrate real linguistic comprehension.

  GPT-4's scores on professional examinations added a different kind of evidence. On the Uniform Bar Examination, it reached above the 90th percentile. On the MATH benchmark of high-school Olympiad problems, it achieved 93% accuracy. It also produced human-level scores on the United States Medical Licensing Examination. A 2022 survey found that 51% of AI professionals believed LLMs could truly understand language with sufficient training data. By 2023, some models had achieved strong results on the Super General Language Understanding Evaluation, or SuperGLUE.

  Kelsey Piper offered a different objection to the stochastic parrot label. The critique, she argued, focuses entirely on pre-training and ignores fine-tuning. Modern LLMs are not only trained to predict the next word; they are also fine-tuned to follow instructions and to favor accurate answers.

  'Shortcut learning' describes a documented counterpoint to high benchmark scores. When human language tests are given to LLMs, false positives can emerge from unrelated statistical patterns in text data. Some models have demonstrated correlations within data that have nothing to do with genuine comprehension. A 2024 Scientific American investigation described a closed workshop at Berkeley. State-of-the-art models there solved novel tier-4 mathematics problems and produced coherent proofs. Such results pointed toward abilities that simple pattern matching could not easily explain.

  1314151617

• 7. Inside the Machine

  ›

  Othello-GPT began as a deliberately limited experiment. A small transformer was trained only on the task of predicting legal moves in the board game Othello. When researchers probed its internal state, they found it had developed a representation of the Othello board itself. Modifying that internal representation caused the model's predictions to change in precisely the correct way. The model was not recalling move sequences; it was tracking the current state of the game.

  A transformer trained on Karel, a simplified programming language, produced similar results. Like the Othello model, it developed an internal representation of the semantics of Karel programs. Modifying that representation produced appropriate changes in the model's outputs. The model also generated correct programs that were, on average, shorter than those in its training data, suggesting it had abstracted beyond memorization.

  Grokking names a phenomenon where a model initially memorizes its training data outputs. After continued training, it abruptly finds a solution that generalizes to entirely new data. The shift is sudden rather than gradual, suggesting an internal reorganization rather than the incremental accumulation of examples.

  Anthropicapplied mechanistic interpretability techniques to Claude, using attribution graphs to trace circuits of reasoning. The research found that Claude 3.5 Haiku employs 'remarkably general abstractions' and processes information through chains of what the researchers described as fuzzy logical inference. It forms 'internally generated plans for its future outputs' and 'works backwards from its longer-term goals.' Anthropic noted that 'the mechanisms of the model can apparently only be faithfully described using an overwhelmingly large causal graph.' The research also identified mechanisms that could underlie a simple form of metacognition. Claude 3.5 Haiku 'thinks about' its own level of knowledge before reaching an answer. That finding was not one the original stochastic parrot paper had anticipated.

  1920212218