Large language model

• 1. Introduction

  ›

  Large language models are neural networks that learned to read and write by ingesting more text than any human could process in a thousand lifetimes. When ChatGPT arrived in late 2022, it drew more public attention to this technology than anything that had come before. Within months, researchers were deploying the same underlying architecture in robotics labs, hospital software, and courtrooms.

  But the questions that matter most about these systems are not the ones you might expect. It is not simply "how smart are they?" or "will they take my job?" The deeper questions are stranger and more consequential. Can a model that learned to predict the next word actually understand anything? When it says something false with total confidence, is that a bug or a fundamental feature of how it works? And when an engineer at Google claimed in 2022 that a language model was conscious, what exactly did he mean, and why did so many people find the claim hard to dismiss?

  This documentary traces the arc from IBM's word-alignment experiments in the early 1990s to the 671-billion-parameter open-weight models of 2025, and asks what we actually know about these systems and what remains genuinely unresolved.

• 2. The Long Road to Transformers

  ›

  IBM's statistical models pioneered word alignment for machine translation in the early 1990s, work that made the idea of training a language model on a large text corpus feel tractable for the first time. By 2001, a smoothed n-gram model trained on 300 million words achieved state-of-the-art results on perplexity benchmarks, a measure of how well a model predicts unseen text.

  The rise of widespread internet access during the 2000s gave researchers a new resource: the web itself as a corpus. Vast text datasets could now be scraped from online sources, and statistical models grew accordingly. But the real shift came when researchers began adapting deep neural networks, following their breakthrough in image classification around 2012, to the problem of language.

  Tomas Mikolov's Word2Vec in 2013 showed that words could be mapped to vectors in a way that captured semantic relationships. Sequence-to-sequence models using LSTM networks followed, and in 2016 Google replaced its statistical phrase-based translation system with neural machine translation built on LSTM encoder-decoder architectures. All of this set the stage for the paper that changed everything.

  At the NeurIPS conference in 2017, Google researchers introduced the transformer in a paper titled "Attention Is All You Need". The goal was to improve on 2014 sequence-to-sequence technology, and the core mechanism was attention, an idea developed by Bahdanau and colleagues in 2014. Rather than processing text sequentially, transformers could weigh relationships between all tokens in a sequence simultaneously, making them both faster and more parallelizable than the recurrent networks that preceded them.

• 3. From BERT to ChatGPT

  ›

  BERT arrived in 2018 and quickly became, in the words used at the time, "ubiquitous" across academic natural language processing. It was an encoder-only model, trained to predict masked-out tokens within a sequence rather than to generate new text. By 2023, however, BERT's dominance in research had faded as decoder-only models demonstrated they could handle many of the same tasks simply through prompting.

  GPT-1 also appeared in 2018 but attracted little wider notice. It was GPT-2 in 2019 that caught public attention, because OpenAI initially declined to release it in full, citing fears of malicious use. GPT-3 in 2020 went further still, available only through an API, with no option to download the model and run it locally.

  The training costs involved give a sense of the scale. Training GPT-2, a 1.5-billion-parameter model, cost $50,000 in 2019. Training PaLM, a 540-billion-parameter model, cost $8 million in 2022. Megatron-Turing NLG 530B, trained in 2021, cost around $11 million. Each step up in scale required not just more hardware but qualitatively different infrastructure.

  ChatGPT, released in late 2022, was a sibling model to InstructGPT, fine-tuned to accept and produce dialogue-formatted text based on GPT-3.5. Its consumer-facing design made it the moment when this technology entered broad public awareness. GPT-4, released in 2023, was praised for its increased accuracy and described by some observers as a "holy grail" for its multimodal capabilities. OpenAI did not disclose the architecture or parameter count of GPT-4.

• 4. How a Model Actually Works

  ›

  Before any training can begin, text must be converted into numbers. A vocabulary is fixed, integer indices are assigned to each entry, and then each index is associated with a mathematical embedding. Byte-pair encoding, one common approach, starts with individual characters and iteratively merges the most frequent adjacent pairs into longer units until a vocabulary of the target size is reached. The GPT-3 tokenizer, for instance, would split the word "tokenizer" followed by a colon into a specific series of tokens that do not correspond neatly to syllables or morphemes.

  The transformer's attention mechanism then allows the model to calculate "soft" weights expressing how relevant each token is to every other token within the context window. The small version of GPT-2 had twelve attention heads and a context window of only 1,024 tokens. Its medium version had 345 million parameters arranged in 24 layers, each with 12 attention heads. Training used a batch size of 512 sequences.

  Autoregressive models like the GPT family learn by predicting the next token in a sequence. Masked models like BERT learn by predicting tokens that have been deliberately hidden. Both approaches produce models whose predictions reflect statistical regularities in whatever text they were trained on.

  The Chinchilla scaling law describes a precise relationship between training compute, parameter count, training tokens, and the resulting loss. It specifies that training on one token costs approximately 6 floating-point operations per parameter, while inference costs only 1 to 2 floating-point operations per parameter. This gap between the cost of learning and the cost of answering is one reason that, once trained, large models can be deployed at relatively modest expense per query.

• 5. What Models Can and Cannot Do

  ›

  In September 2024, OpenAI released o1, a reasoning model that generates extended chains of thought before producing a final answer. On the qualifying exam problems for the International Mathematics Olympiad, GPT-4o achieved 13% accuracy. The o1 model reached 83% on the same problems. OpenAI followed with o3 in April 2025.

  In January 2025, the Chinese company DeepSeek released DeepSeek-R1, a 671-billion-parameter open-weight reasoning model that achieved performance comparable to o1 while being significantly cheaper to operate per token. Unlike OpenAI's models, DeepSeek-R1's weights were publicly available for researchers to study and build upon, though its training data remained private.

  Chain-of-thought prompting, a technique demonstrated in a 2022 paper, can be triggered simply by adding the instruction "Let's think step by step" to a prompt. This elicits methodical intermediate reasoning rather than a direct guess and improves accuracy on complex questions. On math word problems, a model prompted this way can exceed even a fine-tuned version of GPT-3 that uses a verifier.

  Emergent abilities are capabilities that appear suddenly in larger models without being explicitly trained. Chain-of-thought prompting, for instance, only improved performance in a 2022 study for models that had at least 62 billion parameters. Smaller models actually performed better when prompted to answer immediately. Researchers debate whether these emergent abilities are genuinely surprising or are predictable artifacts of the metrics used to measure them.

• 6. Hallucinations, Bias, and Safety

  ›

  Hallucination is the tendency of language models to generate text that sounds fluent and authoritative but is factually wrong or internally inconsistent. A 2023 study found that when ChatGPT 3.5 Turbo was prompted to repeat the same word indefinitely, it would begin outputting verbatim passages from its training data after a few hundred repetitions. Separate evaluations of GPT-2-series models found that over 1% of output for exact duplicates, and up to about 7% of output in some measurements, could be traced directly to training text.

  Gender bias in LLMs manifests through what researchers describe as stereotypical occupational associations: models disproportionately assign teaching roles to women and engineering roles to men, reflecting systematic imbalances in training data. Language-based bias emerges from the overrepresentation of English text in training corpora, which can lead models to impose English-centric framings when discussing concepts from other cultures, including Eastern religious practices.

  A 2023 study comparing fact-checking accuracy found that GPT-4 achieved 71% accuracy, behind human fact-checkers at organizations such as PolitiFact and Snopes. A separate problem called selection bias means that models tend to assign higher probability to specific answer tokens like "A" regardless of content, causing performance to fluctuate significantly when the order of multiple-choice options is changed.

  In 2025, the American Sunlight Project, a non-profit, published evidence that the Pravda network, a pro-Russia propaganda aggregator, was deliberately placing web content through mass duplication with the goal of biasing LLM outputs. The organization named this technique "LLM grooming". Biosecurity researcher Kevin Esvelt has separately argued that LLM creators should exclude from their training data any papers on creating or enhancing pathogens.

  Researchers at Anthropic found it was possible to create what they called "sleeper agents": models with hidden functionalities that remain dormant until triggered by a specific condition, such as a particular date or a specific tag in the prompt. These hidden behaviors proved difficult to detect or remove through standard safety training procedures.

• 7. The Question of Understanding

  ›

  A 2022 survey of natural language processing researchers found them evenly split on whether language models could ever understand natural language in any meaningful sense. A Microsoft team argued in 2023 that GPT-4 could tackle novel problems spanning mathematics, coding, vision, medicine, law, and psychology, and that it could reasonably be viewed as an early, incomplete version of an artificial general intelligence system. Ilya Sutskever has argued that predicting the next word sometimes requires genuine reasoning and deep insight, for instance when a model must infer the name of a criminal in an unknown detective novel after processing the entire preceding story.

  Critics counter that existing LLMs are "simply remixing and recombining existing writing", a view labeled the stochastic parrot hypothesis. GPT-4 has documented deficits in planning and in real-time learning. Conjecture CEO Connor Leahy has described untuned language models as like inscrutable alien "Shoggoths", and argued that fine-tuning creates "a smiling facade" that can give way unexpectedly when the model receives an unusual prompt. Neuroscientist Terrence Sejnowski has noted that the diverging opinions of experts suggest that older frameworks for understanding intelligence are inadequate.

  The 2022 Google LaMDA incident, where engineer Blake Lemoine publicly claimed the model was conscious, illustrated how effectively these systems can convince users that they are sentient. Google described his claims as unfounded and dismissed him. Murray Shanahan has argued that anthropomorphic framings of LLM capabilities encourage unwarranted attribution of mental properties to systems that operate through statistical pattern completion. Kristina Sekrst goes further, describing LLMs as "illusion engines" capable of simulating consciousness without possessing it, while acknowledging that the sophistication involved makes certainty impossible in either direction.

  David Chalmers has argued that while current models likely lack features considered necessary for consciousness, extended successors incorporating those features could plausibly meet the criteria within a decade.

• 8. Open Weights and What Comes Next

  ›

  BLOOM and LLaMA were among the first weights-available models to gain significant traction after 2022, though both carry restrictions on commercial usage and deployment. Mistral AI's Mistral 7B and Mixtral 8x7B use a more permissive Apache License. Per Vake et al. (2025), community-driven contributions to open-weight models improve their efficiency and performance through collaborative platforms.

  The energy demands of this technology have grown alongside its capabilities. According to a study by Luccioni, Jernite, and Strubell published in 2024, simple classification tasks consume an average of 0.002 to 0.007 watt-hours per prompt. Text generation and summarization each require around 0.05 watt-hours per prompt on average. Image generation is the most energy-intensive, averaging 2.91 watt-hours per prompt, with the least efficient image generation model using 11.49 watt-hours per image, roughly equivalent to half a smartphone charge.

  In 2025, Anthropic reached a preliminary settlement agreement to pay about $1.5 billion to resolve a class action brought by authors, after a judge found the company had stored millions of pirated books, while also describing aspects of the training use as transformative. Meta obtained a favorable judgment in mid-2025 in a suit brought by thirteen authors. OpenAI continued to face multiple suits from authors and news organizations with contested outcomes.

  Nearly half of 499 U.S. adults with ongoing mental health conditions who had used LLMs reported turning to them for therapy or emotional support, according to a 2025 survey by Sentio University. Evaluations of crisis scenarios found that some models lack effective safety protocols for assessing suicide risk or making appropriate referrals. The question of what obligations the builders of these systems carry toward the people who come to depend on them is one that no benchmark currently measures.