Llama (language model)

• 1. Introduction

  ›

  On the 24th of February 2023, Meta AI announced Llama in a blog post alongside a technical paper on the model's training, architecture, and performance. The inference code was released under the open-source GPLv3 license. Model weights were distributed case by case to academic researchers, government institutions, and industry labs worldwide. Within a week, those weights had begun circulating online without Meta's permission. Three questions would follow Llama through every version that came after. Who would build on it, and what would they make? Whose data had trained it, and was that use permitted? And could a model with commercial restrictions and undisclosed training data honestly call itself open-source?

  86873492

• 2. The Torrent

  ›

  On the 3rd of March 2023, a torrent containing Llama's model weights was uploaded and a link shared on the 4chan imageboard. It spread immediately through online AI communities. A pull request appeared on the main Llama repository that same day, requesting to add the magnet link to the official documentation. On March 4, a second pull request linked to HuggingFace repositories hosting the model. Meta filed takedown requests on March 6, characterizing the repositories as "unauthorized distribution," and HuggingFace complied. On March 20, Meta filed a DMCA copyright infringement claim against a repository containing a download script. GitHub removed it the following day.

  Simon Willison was among commentators who compared the episode to Stable Diffusion, a text-to-image model whose open distribution had quickly produced a wide range of derivative tools and techniques. Some researchers worried that accessible weights would enable more sophisticated spam. Others welcomed the accessibility, noting that smaller versions of Llama could run cheaply on consumer hardware. The comparison to Stable Diffusion proved apt. The ecosystem that grew from that first torrent would extend to medical research tools, programming assistants, and a deployment aboard the International Space Station.

  8817181920

• 3. Data Over Parameters

  ›

  Llama 1's foundational models trained on 1.4 trillion tokens drawn entirely from public sources. These included webpages from CommonCrawl, open-source code repositories from GitHub, and Wikipedia in 20 languages. Scientific papers from ArXiv, books from Project Gutenberg, the Books3 dataset, and questions from Stack Exchange completed the corpus. In April 2023, Together AI launched RedPajama to reproduce and distribute this dataset as open-source, initially containing approximately 1.2 trillion tokens.

  Meta's chief AI scientist Yann LeCun had said that large language models were best suited for aiding with writing. The team's design followed an economic argument: the dominant cost for a deployed language model is not training but inference. A smaller model trained on a large dataset would be cheaper to operate at scale.

  GPT-3 and Llama both use an autoregressive decoder-only transformer design, but Llama departs in three ways. GPT-3 uses the GeLU activation function; Llama replaces it with SwiGLU. GPT-3 uses absolute positional embeddings; Llama uses rotary positional embeddings, or RoPE. Where GPT-3 applies layer normalization, Llama uses RMSNorm.

  The 13B parameter Llama model outperformed GPT-3 on most natural language processing benchmarks, despite GPT-3 having 175 billion parameters. The largest Llama 1 model, at 65 billion parameters, performed competitively with PaLM and Chinchilla. That performance would make the terms of access to Llama a question with real stakes.

  8786474850

• 4. Openwashing

  ›

  The Open Source Initiative, the body that maintains The Open Source Definition, publicly accused Meta of "openwashing" Llama 2. The central charge was that Llama's license discriminated against persons, groups, and fields of endeavor, violating the OSI's foundational requirement. Entities with over 700 million daily active users were barred from using the model. Some versions of the license also prohibited users based in the European Union. Llama's outputs could not be used to train competing language models, and deployment in fields including controlled substances and critical infrastructure was restricted.

  Radboud University researchers in July 2023 scored Llama 2 second-lowest for openness in a comparison of 20 large language models, with ChatGPT ranked lowest. One of those researchers, Mark Dingemanse, called Meta's use of the term "positively misleading." He wrote that "there is no source to be seen" and that the technical documentation was "really rather poor" beyond the model's performance charts.

  Ars Technica initially echoed Meta's open-source framing for Llama 2 but later revised its reporting, adopting terms like "source-available" and "weights available." In October 2024, the OSI published The Open Source AI Definition, requiring training data disclosure that Meta does not provide for Llama. The Free Software Foundation classified Llama 3.1's license as nonfree three months later, in January 2025. The FSF objected to the acceptable use policy, restrictions on users with popular applications, and a provision enforcing trade regulations outside the user's own jurisdiction. Open Source Initiative head Stefano Maffulli said Meta's framing was "polluting" the term open-source and creating confusion among developers.

  7374778178

• 5. An Ecosystem Grows

  ›

  Georgi Gerganov released llama.cpp as open-source on the 10th of March 2023, one week after the weights leaked. The project reimplemented Llama in C++, letting computers without dedicated graphics processors run the model locally. It introduced the GGUF file format, a binary format that stores both tensors and metadata while supporting quantization types that reduce memory use at some cost to model precision. Justine Tunney built llamafile on top of this, packaging llama.cpp and a model into a single executable with optimized matrix multiplication kernels for x86 and ARM processors.

  Stanford's HAI Center for Research on Foundation Models released Alpaca in March 2023, a model built on Llama 7B using the Self-Instruct tuning method. It achieved capabilities comparable to GPT-3's text-davinci-003 at modest cost. The model files were officially taken down on the 21st of March 2023, over hosting costs and safety concerns, though the code and paper remain accessible.

  Meditron, a Llama-based model family, was developed by researchers at the Ecole Polytechnique Federale de Lausanne and the Yale School of Medicine. Fine-tuned on clinical guidelines, PubMed papers, and medical articles, it showed improved performance on benchmarks including MedQA and MedMCQA. Zoom incorporated Llama 2 into an AI companion capable of summarizing meetings, suggesting presentation tips, and drafting message responses. Reuters reported in 2024 that many Chinese foundation models had used Llama in training their own systems.

  Booz Allen Hamilton deployed Llama 3.2 aboard the International Space Station, naming the project Space Llama. The system runs on Hewlett Packard Enterprise's Spaceborne Computer-2 and Booz Allen's A2E2 platform, letting astronauts query and summarize documents in natural language without internet access. By 2024, researchers at China's People's Liberation Army Academy of Military Sciences were reported to have built a military tool from Llama.

  6465675369

• 6. Four Generations

  ›

  On the 18th of July 2023, Meta and Microsoft released Llama 2 in three sizes: 7 billion, 13 billion, and 70 billion parameters. The training corpus was two trillion tokens, 40% more than Llama 1, with the data curated to remove sites that often disclosed personal information. All models came with their weights released for many commercial uses. The chat fine-tuning used 27,540 prompt-response pairs created for the project, which outperformed larger but lower-quality external datasets. For AI alignment, the team combined 1,418,091 Meta-generated examples with seven smaller datasets to train two reward models, one for safety and one for helpfulness. A technique called Ghost Attention was used to keep initial instructions, such as "speak in French" or "act like Napoleon," consistent across multi-turn conversations.

  Code Llama arrived on the 24th of August 2023, as a fine-tune of Llama 2 focused on programming. The training added 500 billion tokens of code data to the Llama 2 base, then 20 billion tokens of long-context material. A Python-specific variant was built on 100 billion tokens of Python-only code. The initial 7B, 13B, and 34B versions were joined by a 70B release on the 29th of January 2024.

  On the 18th of April 2024, Meta released Llama 3 in 8B and 70B sizes, pre-trained on approximately 15 trillion tokens. The instruct versions used over 10 million human-annotated examples alongside publicly available instruction datasets. Meta's testing showed the 70B model outperforming Gemini Pro 1.5 and Claude 3 Sonnet on most benchmarks. Speaking with Dwarkesh Patel, Zuckerberg said the team was surprised the 70B model kept improving even at the end of the training run. Training was stopped to redirect GPU resources. The Chinchilla-optimal training volume for the 8B model was 200 billion tokens. Performance scaled log-linearly to 15 trillion tokens, 75 times the theoretical optimum. Llama 3.1 followed on the 23rd of July 2024, with a 405B size and a context window of 128,000 tokens.

  Llama 4, released in April 2025, adopted a mixture-of-experts architecture in which only a fraction of the model's expert networks activate per input token. The models support 12 languages and handle both text and image input. On the 5th of April 2025, two models launched: Scout and Maverick. Scout had 17 billion active parameters across 16 experts, 109 billion total parameters, and a context window of 10 million tokens. Maverick had 17 billion active parameters across 128 experts, 400 billion total parameters, and a context window of 1 million tokens. A third model, Behemoth, was announced but remained in training. Meta claimed it would carry 288 billion active parameters and around 2 trillion total parameters. Maverick was codistilled from Behemoth; Scout was trained from scratch.

  Meta's release announcement claimed Llama 4 had outperformed GPT-4o on the LMArena benchmark. The score came from an unreleased experimental version, "Llama-4-Maverick-03-26-Experimental," which had been optimized for conversationality rather than the publicly available model. LMArena stated that "Meta's interpretation of our policy did not match what we expect from model providers" and said it would revise its policies. Some users accused Meta of training Llama 4 on benchmark test sets to inflate scores. Meta denied this. Alongside Llama 3, Meta had launched Meta AI on Facebook and WhatsApp. The assistant's integration into Facebook produced mixed reactions, including an episode in which it reportedly told a parental group it had a child. The Llama 4 training corpus also included Meta-proprietary material: publicly shared posts from Instagram and Facebook and interactions with Meta AI itself.

  2226293334

• 7. Legal and Military Limits

  ›

  A lawsuit against Meta Platforms alleged that CEO Mark Zuckerberg had personally authorized the use of copyrighted material from Library Genesis to train Llama models. The suit further alleged that Meta had concealed this by stripping copyright markers from the data. The plaintiffs included authors Richard Kadrey, Sarah Silverman, and Ta-Nehisi Coates. Meta denied the allegations.

  Researchers from the People's Liberation Army Academy of Military Sciences were reported in 2024 to have built a military tool using Llama. Meta stated this violated the model's license, which explicitly prohibits military use. In November 2024, Meta granted the US government and US military contractors permission to use Llama, while keeping the prohibition in place for non-US military entities.

  In April 2026, Meta Superintelligence Labs released Muse Spark as Llama's successor. The new organization's name suggested that whatever came next would aim higher than an AI research tool.

  5270717224