EleutherAI

• 1. Introduction

  ›

  EleutherAI began on the 7th of July, 2020, not as an institute or a company, but as a Discord server with the tentative name "LibreAI." Connor Leahy, Leo Gao, and Sid Black built it around a single goal. They wanted to create an open-source version of GPT-3, the language model OpenAI had introduced to the world but refused to release publicly. The server renamed itself EleutherAI within the month, borrowing from eleutheria, the Greek word for liberty. What does it take for a volunteer community to produce models and research that larger institutions worldwide would come to rely on? And why did the same group later face accusations of theft from creators whose work had appeared in a training dataset?

  163

• 2. A Ladder of Models

  ›

  The GPT-Neo series launched on the 21st of March, 2021, with model sizes at 125 million, 1.3 billion, and 2.7 billion parameters. At the time of release, the largest was the biggest open-source model in the GPT-3 style available anywhere in the world. EleutherAI distributed the models under the Apache 2.0 free software license. That permissive choice let companies incorporate them into commercial products at no licensing cost. These releases were later described as having "fueled an entirely new wave of startups."

  GPT-J-6B followed on the 9th of June, 2021. At six billion parameters, it again claimed the title of the world's largest publicly available GPT-3-style model.

  Google's TPU Research Cloud Program was EleutherAI's main source of computing in its first year. The group had initially turned down outside funding, preferring to stay independent. By early 2021, that position had changed. CoreWeave, a cloud computing company, and SpellML, a cloud infrastructure company, both contributed access to GPU clusters large enough for large-scale machine learning work. On the 10th of February, 2022, EleutherAI released GPT-NeoX-20B, a model at twenty billion parameters built on the capacity CoreWeave had provided. Before any of these models went public, EleutherAI had already released a training dataset that other teams would draw on just as heavily. That collection had been assembled on the last day of 2020.

  781617

• 3. The Pile

  ›

  Microsoft's Megatron-Turing Natural Language Generation, one of the largest language models of its era, was trained on a dataset called The Pile. That adoption reflected how broadly useful The Pile had become outside EleutherAI's own projects. The dataset contained 886 gigabytes of text, drawing from books, documentary subtitles, movie and television transcripts, and material from online video platforms.

  The Pile stood out from comparable collections in one key respect: documentation. Researchers at EleutherAI published a full account of what the dataset contained and why each source had been chosen. No comparable training dataset of the era came with that degree of transparency about its contents.

  In July 2024, Proof News and Wired published an investigation into The Pile. It found subtitles from more than 170,000 YouTube videos, spread across over 48,000 channels. YouTubers and other creators accused EleutherAI of theft. Common Pile v0.1, the dataset EleutherAI would build in direct response, required partnerships with a large number of collaborators. It would not arrive until June 2025.

  172121318

• 4. Images From Text

  ›

  OpenAI announced DALL-E in January 2021, a model capable of generating images from text descriptions. When OpenAI chose not to release DALL-E for public use, Katherine Crowson and digital artist Ryan Murdock set out to find an alternative. Both were working within the EleutherAI community.

  CLIP, a model OpenAI had already released for mapping relationships between text and images, was one of the two components they combined. The other was VQGAN, a generative model for producing images. Building on earlier ideas from Google's DeepDream, Crowson and Murdock found that running CLIP's understanding through VQGAN's generation process could create images from text prompts. The resulting technique became known as VQGAN-CLIP.

  Crowson published the method on Twitter, posting notebooks anyone could run for free without specialized hardware. The structural question of how a volunteer organization could sustain that kind of open-access research at scale would take until early 2023 to resolve.

  2633283234

• 5. Becoming an Institute

  ›

  Early 2023 brought EleutherAI's most significant structural change. The group incorporated formally as a non-profit research institute, with Stella Biderman, Curtis Huebner, and Shivanshu Purohit taking responsibility for running it. Along with the new structure came a deliberate shift in research priorities.

  EleutherAI acknowledged that "there is substantially more interest in training and releasing LLMs than there once was." That recognition gave the organization room to redirect its work. Interpretability, alignment, and scientific research became the new focus: how language models work internally, and whether they can be made safer and more reliable. By 2025, that shift toward scientific research had drawn EleutherAI into collaborations with government institutions examining the safety implications of how AI models are trained.

  101154

• 6. Common Pile

  ›

  Every piece of content in Common Pile v0.1 came from sources whose licenses explicitly permitted AI training. EleutherAI trained two models from the new dataset, a demonstration that building from only licensed material was viable at scale.

  Working with the United Kingdom's AI Security Institute, EleutherAI produced a finding with implications beyond Common Pile itself. Their research showed that filtering a dataset to remove specific concepts can maintain a model's overall performance. It also reduces the model's capacity to provide harmful information. The collaboration with the UK's AI Security Institute marked the point where EleutherAI's training data research became a direct resource for national-level AI safety evaluation.

  13141524