EleutherAI

• 1. The Discord Birth

  ›

  On the 7th of July 2020, a group of three programmers gathered inside a private Discord server to discuss artificial intelligence. Connor Leahy, Leo Gao, and Sid Black wrote the initial code for what they first called LibreAI. They intended to build an open-source version of OpenAI's powerful GPT-3 model without corporate restrictions. The name changed to EleutherAI later that month, drawing from the Greek word for liberty. This small online gathering would eventually become a formal non-profit research institute by early 2023.

  13

• 2. The Pile Dataset

  ›

  EleutherAI released The Pile on the 31st of December 2020 as an 886 GB dataset designed for training large language models. Researchers curated this diverse text collection to contain information they believed AI should learn. It became widely used to train other systems including Microsoft's Megatron-Turing Natural Language Generation. The dataset contained copyrighted material such as books and subtitles from over 170,000 YouTube videos across more than 48,000 channels. An investigation by Proof news in July 2024 exposed these contents and drew accusations of theft from content creators.

  213

• 3. Gpt Model Evolution

  ›

  The team released their first major model series in March 2021 with sizes ranging from 125 million to 6 billion parameters. GPT-J-6B arrived on the 9th of June 2021 as the largest open-source GPT-3-like model available at that time. These models operated under the Apache 2.0 free software license and fueled a new wave of startups. On the 10th of February 2022, EleutherAI launched GPT-NeoX-20B using resources provided by CoreWeave. This scaled-up version demonstrated how commercial cloud infrastructure could support massive machine learning research projects.

  7825

• 4. Text To Image Breakthroughs

  ›

  Katherine Crowson and Ryan Murdock began developing text-to-image synthesis models following OpenAI's January 2021 release of DALL-E. They combined CLIP technology with VQGAN to create the VQGAN-CLIP system without requiring special equipment. Crowson shared her work through public tweets containing notebooks that anyone could run for free. Their approach built upon earlier ideas from Google's DeepDream project. The resulting images showed how AI could generate art based solely on text descriptions.

  262833

• 5. Cloud Funding Shift

  ›

  EleutherAI initially relied on Google's TPU Research Cloud Program to source their computing power. By early 2021 they accepted funding from CoreWeave and SpellML in exchange for access to powerful GPU clusters. This shift allowed them to scale up their research capabilities significantly. The organization later incorporated as a non-profit institute run by Stella Biderman, Curtis Huebner, and Shivanshu Purohit. Commercial partnerships became necessary to maintain large-scale machine learning operations.

  4511

• 6. Ethical Policy Changes

  ›

  In June 2025 EleutherAI released Common Pile v0.1 which contained only works permitting use for training AI models. This new dataset removed controversial copyrighted material found in previous releases. Collaboration with the UK's AI Security Institute helped filter training data to reduce harmful information output. Researchers discovered that removing key concepts maintained performance while improving safety standards. These policy shifts reflected growing concerns about copyright infringement and ethical AI development practices.

  131415