When was GPT-3 published by OpenAI?
GPT-3 was published on the 28th of May 2020. A group of thirty-one engineers and researchers at OpenAI released an arXiv preprint describing the third-generation language model on that date.
Questions about GPT-3
Short answers, pulled from the story.
How many parameters does GPT-3 have and what is its context window size?
GPT-3 contains 175 billion parameters with a context window of 2048 tokens. Each parameter required 16-bit precision consuming 350GB of storage space alone while allowing the system to process substantial blocks of text before losing track of earlier information.
What training data sources were used for GPT-3?
Sixty percent of the weighted pre-training dataset came from Common Crawl containing 410 billion byte-pair-encoded tokens. The remaining data included WebText2, Books1, Books2, and Wikipedia which together provided hundreds of billions of words scraped from sixty million domains over twelve years.
Who licensed GPT-3 exclusively in September 2020?
Microsoft announced on the 22nd of September 2020 that it had licensed GPT-3 exclusively for its own products and services. This multi-billion dollar investment permits OpenAI to offer a public-facing API while only Microsoft gains access to the underlying source code.
Why did Noam Chomsky criticize GPT-3's scientific value?
Noam Chomsky expressed skepticism stating that GPT-3 works just as well for impossible languages as for actual languages. He argued the system is refuted by normal scientific criteria if intended as a true language model because it tells us nothing about language or cognition generally.