Generative pre-trained transformer

• 1. Introduction

  ›

  The generative pre-trained transformer is the architecture behind ChatGPT and most of the AI chatbots that followed it. On the 30th of November 2022, when ChatGPT launched, millions of people encountered one for the first time. They typed questions into a text box and received fluent, detailed responses. The technology powering that exchange had been years in development, built on a design that had only recently been published. How does a model trained to predict the next word in a sequence end up capable of writing essays, explaining concepts, and holding a conversation? And why did OpenAI spend 2023 trying to claim legal ownership of the very word that names this class of technology?

  515

• 2. Attention Is All You Need

  ›

  'Attention Is All You Need' was published by Google researchers in 2017, and the claim turned out to be accurate. Before it appeared, the leading approach to processing language with neural networks was the recurrent neural network, which read text word by word, in sequence. That sequential structure had a hard ceiling. A model working through a long document in strict order was slow to train and struggled to connect ideas that appeared far apart in the text. The transformer solved this with an attention mechanism. Rather than moving through text word by word, a transformer reads an entire passage at once and calculates how every word relates to every other word simultaneously. Since 2017, systems built on this architecture have been capable of processing, summarizing, contrasting, and answering questions from text across many different domains. The question that followed the paper was which research team would be first to train one large enough to matter.

  359

• 3. BookCorpus and Fine-Tuning

  ›

  On the 11th of June 2018, OpenAI researchers published 'Improving Language Understanding by Generative Pre-Training,' which introduced the first GPT model. The approach had two stages. First, the model was trained on BookCorpus, a large and varied text dataset, with no human labels, learning purely to predict what came next in a sequence. Then it was fine-tuned for specific tasks using a smaller, labeled dataset. This two-stage design bypassed the main obstacle that had slowed earlier language models: supervised training on hand-labeled data required enormous human effort, making serious scale too expensive to attempt. GPT models use only the decoder half of the transformer architecture, pre-trained to predict the next token in a sequence. That task requires no human annotation at all. OpenAI introduced GPT-2 on the 14th of February 2019, scaled to ten times GPT-1's parameter count and dataset size, and with it came a concern the company did not expect.

  1368

• 4. Three Records in Two Years

  ›

  GPT-2 held 1.5 billion parameters and was trained on WebText, a dataset of 40 gigabytes built from 8 million web pages. OpenAI chose not to release the full model at once, citing the risk of malicious misuse. It described the strategy as a 'staged release,' publishing smaller versions first before the full model became available in November. On the 10th of February 2020, Microsoft entered the competition with its Turing Natural Language Generation model. Microsoft described it as the largest language model ever published, at 17 billion parameters, and reported that it outperformed all previous models at summarization and question answering. That record did not last long. On the 28th of May 2020, OpenAI released GPT-3, with 175 billion parameters trained on a dataset larger than WebText. With only a handful of examples in the prompt, GPT-3 could carry out tasks it had never explicitly trained for. That quality pointed toward new methods for directing what large models could do.

  1617181920

• 5. InstructGPT and the Chatbot Era

  ›

  After GPT-3, OpenAI developed reinforcement learning from human feedback, or RLHF, as a method for aligning model behavior with what users wanted. In January 2022, OpenAI introduced InstructGPT, a series of models fine-tuned from GPT-3 using a combination of supervised training and RLHF. Compared to raw base models, InstructGPT showed higher accuracy and produced less toxic output. OpenAI then refined InstructGPT into a chatbot. For that chatbot, human AI trainers provided conversations in which they played both the user and the AI. That dialogue data was used to train the model. The result was ChatGPT, based on GPT-3.5. GPT-4 arrived on the 14th of March 2023, and was integrated into Microsoft Copilot, GitHub Copilot, Snapchat, Khan Academy, and Duolingo. The wider industry responded. EleutherAI had released GPT-J, an open-weight model, in 2021. Google developed PaLM and then Gemini. Meta AI built Llama. On the 7th of August 2025, OpenAI released GPT-5. The model uses a built-in router to choose automatically between a faster standard model and a slower reasoning model depending on the task.

  24232526

• 6. Who Owns the Name?

  ›

  OpenAI asserted in 2023 that 'GPT' should be treated as an OpenAI brand. In April 2023, the company updated the terms of service for its API to prohibit other businesses from naming their products with 'GPT.' That same month, the United States Patent and Trademark Office declined OpenAI's request to expedite its trademark application. In May 2023, OpenAI hired a brand management service to notify API customers of the naming policy. Those notices stopped short of legal threats such as infringement claims or cease-and-desist demands. Also in May, the USPTO found 'GPT' to be both descriptive and generic, a determination OpenAI was still contesting as of November 2023. The European Union Intellectual Property Office registered 'GPT' as an OpenAI trademark in spring 2023. That registration has been under challenge since spring 2024 and is pending cancellation. Switzerland also registered the trademark for OpenAI in spring 2023. Separately, OpenAI began allowing ChatGPT Plus subscribers to create custom products named GPTs, even as its own terms of service called that naming 'discouraged.'

  4448495247

• 7. Scale, Emergence, and Cost

  ›

  BigBird, Reformer, and FlashAttention are among the systems researchers built to address a fundamental tension in transformer scaling. As models grow larger and handle longer sequences, memory and computation requirements expand sharply. The standard attention mechanism has a quadratic relationship to input length, meaning costs rise sharply as sequences grow longer. These newer approaches introduced sparse attention patterns and memory-efficient computation to support longer contexts at lower cost. Certain capabilities of large GPT models do not appear gradually as models grow. Multi-step reasoning and in-context learning are among the abilities that are absent below certain thresholds of size, training data, and compute. Research suggests these capabilities do not scale linearly. They appear at a threshold and are absent below it. That pattern has complicated evaluation: a benchmark may show no improvement across many model generations and then reveal a sudden jump at scale. Training models to reach those thresholds carries significant environmental costs. Large-scale training requires substantial computational resources and contributes to measurable increases in energy consumption. Researchers and observers have called for more efficient training methods and greater transparency in reporting how much compute a model required to build.

  3637383942