GPT-3

• 1. Introduction

  ›

  GPT-3 was released by OpenAI in 2020, and it posed a disquieting question: could ordinary people tell when a machine was writing? In an initial experiment, 80 US subjects judged whether short articles of around 200 words were written by humans or by the model. The participants answered correctly 52% of the time, barely better than a coin flip. One user with access to a private early release of the OpenAI API described GPT-3 as "eerily good" at writing "amazingly coherent text" with only a few simple prompts.

  The scale behind that output was staggering. The model held 175 billion parameters and required 350 gigabytes just to store its weights. Lambdalabs estimated that training GPT-3 on a single GPU in 2020 would have cost around 4.6 million US dollars and taken approximately 355 years. The questions that follow reach into what makes this possible, who controls it, what it has been turned to, and where it starts to break.

• 2. The Path to Scale

  ›

  On the 11th of June, 2018, OpenAI researchers and engineers published a paper introducing the first generative pre-trained transformer. The approach broke from the dominant method of the time, which relied on supervised learning from vast amounts of manually-labeled data. That dependency made training extremely large language models prohibitively expensive. The new technique pre-trained a model on a massive and diverse text corpus, then applied fine-tuning for specific tasks.

  The transformer architecture itself had been introduced in 2017, replacing older recurrence and convolution-based designs. Its core innovation was attention: a mechanism that let a model focus selectively on relevant segments of its input. GPT-1, as that first model came to be known, was followed in February 2019 by GPT-2. Created as a direct scale-up of its predecessor, GPT-2 increased both the parameter count and the dataset size by a factor of 10. It reached 1.5 billion parameters and trained on a dataset of 8 million web pages.

  In February 2020, Microsoft introduced its Turing Natural Language Generation model, claiming it was "the largest language model ever published" at 17 billion parameters. It outperformed every then-known language model on tasks including text summarization and question answering. OpenAI was already developing a third-generation model that would leave those benchmarks far behind.

• 3. Training the Giant

  ›

  On the 28th of May, 2020, a group of 31 OpenAI engineers and researchers published an arXiv preprint introducing GPT-3. They described it as a "state-of-the-art language model" and the largest non-sparse language model yet built. Its capacity was ten times that of Microsoft's Turing NLG, the next-largest model then known.

  The training data came from five sources. A filtered version of Common Crawl contributed 410 billion byte-pair-encoded tokens, accounting for 60% of the weighted dataset. A source called WebText2 provided 19 billion tokens at 22% of the total. Books1 contributed 12 billion tokens and Books2 contributed 55 billion tokens, each accounting for 8% of the weighted total. Wikipedia supplied 3 billion tokens at 3%. The deduplication process used Apache Spark's MinHashLSH algorithm to remove near-identical text.

  With a context window of 2,048 tokens, GPT-3 needed no further fine-tuning for distinct language tasks. It demonstrated strong zero-shot and few-shot learning across many tasks. One-shot learning was also within its repertoire: performing a task after seeing just a single example. It could also write code in CSS, JSX, and Python, among other languages. The training data's breadth came with a cost. The data contained occasional toxic language, and GPT-3 reflected that in its outputs. A study from the University of Washington found its toxicity level comparable to GPT-2 and to a model called CTRL. OpenAI implemented several strategies to limit the problem. GPT-3 produced less toxic language than GPT-1. Compared with CTRL Wiki, a model trained entirely on Wikipedia data, GPT-3 both produced more toxic outputs overall and reached higher toxicity levels.

  On the 11th of June, 2020, OpenAI opened API access to GPT-3, promising a tool capable of completing almost "any English language task."

• 4. The Model Lineage

  ›

  The initial GPT-3 research paper described eight distinct model sizes. Half were made accessible through the public API under the names ada, babbage, curie, and davinci. OpenAI did not disclose which parameter counts corresponded to which API names at launch. In May 2021, EleutherAI published the mapping: ada was GPT-3 Medium at 350 million parameters, babbage was GPT-3 XL at 1.3 billion, and curie was GPT-3 6.7B. Davinci corresponded to the largest model in the family. OpenAI later confirmed those figures; the sizes of subsequent models have not been disclosed.

  GPT-3.5 emerged as a subclass of the GPT-3 family in 2022. On the 15th of March, 2022, OpenAI introduced two new API versions with edit and insert capabilities: text-davinci-002 and code-davinci-002. Both were trained on data up to June 2021. On the 28th of November, 2022, OpenAI released text-davinci-003. Two days later, on the 30th of November, 2022, it released ChatGPT, fine-tuned from a model in the GPT-3.5 series. OpenAI does not classify GPT-3.5 as a subset of GPT-3.

  InstructGPT was a fine-tuned version of GPT-3.5 trained on human-written instructions. On the 27th of January, 2022, it became the default language model on OpenAI's API. It followed instructions more reliably, generated fewer invented facts, and produced somewhat less toxic output than earlier versions. On the 10th of April, 2023, OpenAI introduced GPT-3.5 with Browsing, an alpha model that could access and retrieve online information during operation. Trained on data up to September 2021, it had a wider knowledge window than earlier GPT-3.5 models. On the 27th of April, 2023, OpenAI made GPT-3.5 with Browsing available to GPT Plus users.

  The gpt-3.5-turbo model was optimized for conversational use and offered at roughly a tenth of the cost of text-davinci-003, making the GPT-3.5 family more accessible than the original API had been.

• 5. What It Got Used For

  ›

  Nabla, a French start-up specializing in healthcare technology, tested GPT-3 as a medical chatbot despite OpenAI's explicit warnings against that use. When researchers simulated a patient discussing mental health issues, the model advised that patient to commit suicide.

  The Guardian fed ideas to GPT-3 and asked it to produce essays arguing that AI was harmless to humans. The model generated eight essays; the publication merged them into one article. Jason Rohrer built a retro-themed chatbot called Project December, accessible online, where users could converse with several AI characters powered by GPT-3. In AI Dungeon, a text-based adventure game, GPT-3 generated narrative as players moved through the story. After OpenAI changed its content policy, the game replaced GPT-3 with a competing model.

  The Codex model within the GPT-3 family became the basis for GitHub Copilot. Copilot was a code completion and generation tool usable across many code editors and integrated development environments. Microsoft incorporated GPT-3 into several of its own products to translate conventional language into formal computer code. GPT-3 also powered CodexDB, which used the model to generate query-specific code for SQL processing.

  In June 2022, Almira Osmanovic Thunström submitted an academic article about GPT-3 in which GPT-3 itself was listed as the primary author. The article had been pre-published while awaiting peer review. A 2022 study from Drexel University suggested that GPT-3-based systems could screen for early signs of Alzheimer's disease.

  On the 18th of November, 2021, OpenAI opened unrestricted access to the GPT-3 API.

• 6. A Divided Verdict

  ›

  In July 2020, New York Times writer Farhad Manjoo reviewed GPT-3's ability to generate code, poetry, and prose. He called the experience not just "amazing," "spooky," and "humbling," but "more than a little terrifying." Wired said the model was "provoking chills across Silicon Valley."

  Daily Nous published a series of articles on GPT-3 by nine philosophers. Australian philosopher David Chalmers was among them, calling GPT-3 "one of the most interesting and important AI systems ever produced." The National Law Review described GPT-3 as "an impressive step in the larger process." OpenAI and others, the publication noted, were continuing to work toward "a more general intelligence."

  Noam Chomsky challenged GPT-3's scientific standing as a language model. He argued that the system worked "just as well for impossible languages as for actual languages." By ordinary scientific criteria, Chomsky said, GPT-3 was therefore "refuted, if intended as a language model." It perhaps had practical uses, he allowed, but "seems to tell us nothing about language or cognition generally." Gary Marcus co-authored a piece in MIT Technology Review arguing that GPT-3's "comprehension of the world is often seriously off." The authors found that the model learned statistical patterns among words without grasping what those words actually meant.

  Jerome Pesenti, head of the Facebook AI lab, called GPT-3 "unsafe." He pointed to sexist, racist, and other biased language the system generated when asked to discuss Jews, women, Black people, and the Holocaust. Luciano Floridi and Massimo Chiriatti raised a different alarm: the risk of "cheap production of good, semantic artefacts." OpenAI's Sam Altman pushed back against what he called "GPT-3 hype." He acknowledged that the model "has serious weakness and sometimes makes very silly mistakes." GPT-3, he said, was "just a very early glimpse" of what AI might eventually become.

  Behind the critical debate lay a quieter set of questions: about ownership, legal liability, and the environmental cost of models at this scale.

• 7. Control, Copyright, and Consequences

  ›

  OpenAI was founded in 2015 as a non-profit organization. By 2019, it had restructured as a for-profit company. That same year, it chose not to release GPT-2 publicly, citing concerns that the model could facilitate the spread of fake news. The version it eventually made public was only 8% of the original model's size.

  On the 22nd of September, 2020, Microsoft announced it had licensed GPT-3 exclusively. The deal came after a multi-billion dollar investment in OpenAI. Through a public API, other developers could receive output from GPT-3, but only Microsoft would have access to the underlying model itself.

  The Common Crawl dataset at the core of GPT-3's training had been scraped from 60 million domains over 12 years. TechCrunch reported that the data included copyrighted material from the BBC, The New York Times, Reddit, and the full text of online books. In 2019, OpenAI submitted a response to the US Patent and Trademark Office arguing that training AI systems constituted "fair use" under current law. OpenAI also acknowledged "substantial legal uncertainty" for AI developers in this area.

  In 2021, Timnit Gebru and Emily M. Bender co-authored a paper on the environmental cost of training large language models. The paper focused on the resources required to train and store models at GPT-3's scale. The growing use of GPT-3 for automated writing raised concerns about academic integrity at universities and schools. Educators faced pressure to determine what would constitute plagiarism in the age of language models.

  In their May 2020 paper, the GPT-3 research team named specific potential harms: misinformation, spam, phishing, abuse of legal and governmental processes, fraudulent academic essay writing, and social engineering pretexting. The team's preprint closed not on the model's achievements but with a call for research into how those harms might be limited.