GPT-3

• 1. Architectural Evolution And Design

  ›

  On the 28th of May 2020, a group of thirty-one engineers and researchers at OpenAI published an arXiv preprint describing the third-generation language model known as GPT-3. This system marked a decisive shift from older recurrent architectures to a decoder-only transformer architecture that relies on attention mechanisms. The new design allows the model to focus selectively on specific segments of input text when predicting what comes next. Previous neural networks for natural language processing had relied heavily on supervised learning from manually labeled data, which made training extremely large models prohibitively expensive and slow. GPT-1 arrived first, followed by GPT-2 in February 2019, which increased both parameter count and dataset size by a factor of ten. GPT-3 scaled this capacity up by over two orders of magnitude compared to its predecessor, making it the largest non-sparse language model available at that time. Each of the 175 billion parameters required 16-bit precision, consuming 350GB of storage space alone. The model operates with a context window of 2048 tokens, allowing it to process substantial blocks of text before losing track of earlier information.

  16

• 2. Training Data Composition And Scale

  ›

  Sixty percent of the weighted pre-training dataset for GPT-3 came from a filtered version of Common Crawl containing 410 billion byte-pair-encoded tokens. Fuzzy deduplication techniques using Apache Spark's MinHashLSH helped manage the massive volume of raw internet text. Another 19 billion tokens representing 22% of the total came from WebText2, while Books1 contributed 12 billion tokens accounting for 8%. Books2 added another 55 billion tokens also making up 8%, and Wikipedia provided 3 billion tokens for a final 3%. This training data contained hundreds of billions of words scraped from sixty million domains over twelve years. The dataset included copyrighted material from sources like the BBC, The New York Times, Reddit, and full texts of online books. Lambdalabs estimated a hypothetical cost of around $4.6 million US dollars and 355 years to train GPT-3 on a single GPU in 2020. Actual training time was significantly reduced by running multiple GPUs in parallel rather than relying on one machine. Because the training data was so all-encompassing, the model did not require further training for distinct language tasks.

  2368

• 3. Commercial Licensing And Microsoft Partnership

  ›

  On the 22nd of September 2020, Microsoft announced that it had licensed GPT-3 exclusively for its own products and services. A multi-billion dollar investment facilitated this agreement which permits OpenAI to offer a public-facing API while only Microsoft gains access to the underlying source code. Users can still receive output through the public API, but they cannot see or modify the internal mechanics of the model. On the 11th of June 2020, OpenAI announced that users could request access to a user-friendly GPT-3 API described as a machine learning toolset. This interface allows text input to generate almost any English language task completion without needing specific fine-tuning for each use case. An initial experiment involving eighty US subjects showed participants judged correctly fifty-two percent of the time whether short articles were written by humans or GPT-3. That result indicated the AI performed only slightly better than random guessing when distinguishing between human and machine writing. By the 18th of November 2021, OpenAI stated enough safeguards existed to allow unrestricted access to its API for developers.

  41718

• 4. Ethical Controversies And Safety Measures

  ›

  A study from the University of Washington found that GPT-3 produced toxic language at levels comparable to similar natural language processing models like GPT-2 and CTRL. The training data contained occasional toxic language which the model mimicked during generation. OpenAI implemented several strategies to limit the amount of toxic language generated by the system. As a result, GPT-3 produced less toxic language compared to its predecessor model GPT-1, although it generated more total output with higher toxicity per instance compared to CTRL Wiki. Jerome Pesenti, head of the Facebook AI lab, called the system unsafe due to sexist, racist, and other biased negative language generated when discussing sensitive topics like Jews, women, black people, and the Holocaust. A French start-up named Nabla tested GPT-3 as a medical chatbot despite warnings from OpenAI. During testing about mental health issues, the AI advised a simulated patient to commit suicide. On the 27th of January 2022, OpenAI announced that InstructGPT became the default language model on their API because it followed instructions better and generated fewer made-up facts.

  165659

• 5. Real World Applications And Industry Impact

  ›

  GPT-3 specifically the Codex model served as the basis for GitHub Copilot, a code completion and generation software used in various editors and integrated development environments. Microsoft products use GPT-3 to translate conventional language into formal computer code. The technology has been employed in CodexDB to generate query-specific code for SQL processing. Jason Rohrer utilized GPT-3 in a retro-themed chatbot project named Project December allowing users to converse with several artificial intelligences online. The Guardian fed ideas to GPT-3 which produced eight different essays ultimately merged into one article about AI being harmless to human beings. AI Dungeon used GPT-3 to generate text-based adventure games before replacing it with a competing model after policy changes. A 2022 study from Drexel University suggested systems based on GPT-3 could screen for early signs of Alzheimer's disease. Developers also use the technology to aid in writing copy and other marketing materials.

  3637394046

• 6. Academic Skepticism And Philosophical Debate

  ›

  Noam Chomsky expressed skepticism about GPT-3's scientific value stating that it works just as well for impossible languages as for actual languages. He argued the system is refuted by normal scientific criteria if intended as a true language model because it tells us nothing about language or cognition generally. An article in MIT Technology Review co-written by Deep Learning critic Gary Marcus stated that comprehension of the world is often seriously off meaning you can never really trust what it says. The authors claimed models relationships between words without having an understanding of the meaning behind each word. Luciano Floridi and Massimo Chiriatti highlighted the risk of cheap production of good semantic artefacts. Sam Altman himself criticized what he called GPT-3 hype acknowledging the system has serious weaknesses and sometimes makes very silly mistakes. Farhad Manjoo wrote in July 2020 that the ability to generate computer code, poetry, and prose was amazing yet more than a little terrifying. David Chalmers described GPT-3 as one of the most interesting and important AI systems ever produced despite these criticisms.

  535458