Llama (language model)

• 1. The February 24th Announcement

  ›

  On the 24th of February 2023, Meta AI published a blog post and research paper that introduced the first version of Llama. The announcement revealed a model family ranging from 1 billion to 65 billion parameters. Access to these weights was not immediate for everyone. Researchers had to apply through an application process to receive permission on a case-by-case basis. This initial release targeted academic researchers, government affiliates, civil society groups, and industry laboratories worldwide. The inference code ran under the GPLv3 license, but the core weights remained restricted. Meta claimed the 13 billion parameter version outperformed GPT-3 despite having only one-tenth the size. The largest 65 billion model competed with state-of-the-art systems like PaLM and Chinchilla. Training relied solely on publicly available information to ensure broad accessibility across different hardware setups.

  348591

• 2. The March Torrent And Legal Response

  ›

  A torrent containing the full weights of the original Llama appeared online on the 3rd of March 2023. A link to this file spread rapidly through communities on the 4chan imageboard. That same day, a pull request requested adding the magnet link to official documentation. By the 4th of March, another pull request added links to HuggingFace repositories hosting the leaked data. Meta filed takedown requests against HuggingFace on the 6th of March, labeling it unauthorized distribution. GitHub complied with a DMCA request from Meta on the 7th of March regarding a script that downloaded the model from mirrors. Reactions to the leak were divided. Some feared malicious uses like sophisticated spam campaigns. Others celebrated the sudden availability for running smaller models cheaply. Commentator Simon Willison compared the event to the release of Stable Diffusion, noting how open distribution spurred rapid tool development. The incident highlighted tensions between restricted academic access and community-driven innovation.

  516171819

• 3. Scaling Through Successive Generations

  ›

  Meta announced Llama 2 on the 18th of July 2023 in partnership with Microsoft. This version introduced three sizes: 7 billion, 13 billion, and 70 billion parameters. Training used 40% more data than the first iteration while keeping the architecture largely unchanged. Code Llama followed on the 24th of August 2023 with versions up to 34 billion parameters. A larger 70 billion version arrived on the 29th of January 2024. On the 18th of April 2024, Meta released Llama 3 with 8 billion and 70 billion parameter options. These models trained on approximately 15 trillion tokens of text. Performance continued scaling log-linearly even when exceeding Chinchilla-optimal dataset sizes. Llama 3.1 launched on the 23rd of July 2024 adding a 405 billion parameter model. The series expanded further with Llama 4 arriving in April 2025. This latest generation shifted to a mixture of experts architecture. It introduced multimodal capabilities handling both text and image inputs across twelve languages.

  2122252627

• 4. Architectural Choices And Data Composition

  ›

  The Llama series employs autoregressive decoder-only transformers similar to GPT-3 but with distinct technical differences. SwiGLU activation functions replaced GeLU choices found in earlier systems. Rotary positional embeddings substituted absolute positional embedding methods. RMSNorm layers took the place of standard layer normalization. Training focused heavily on increasing data volume rather than just parameter counts. The foundational models for Llama 1 used datasets totaling 1.4 trillion tokens from sources like CommonCrawl web scrapes and GitHub repositories. Wikipedia entries in twenty languages formed part of this corpus. Project Gutenberg public domain books and Books3 datasets contributed additional material. ArXiv scientific paper source code and Stack Exchange questions rounded out the training mix. Llama 2 refined this approach by curating data to remove sites disclosing personal information. It upscaled trustworthy sources and fine-tuned chat models using over 27 thousand prompt-response pairs created specifically for the project. Reinforcement learning with human feedback combined Meta examples with seven smaller external datasets to align safety and helpfulness.

  4748495086

• 5. Open Source Debates And Licensing Conflicts

  ›

  Meta described early versions as open-source while restricting commercial use through non-commercial licenses. The Open Source Initiative disputed this classification, stating licenses violated provisions prohibiting discrimination against specific groups or fields. Restrictions blocked usage in controlled substances, critical infrastructure, and by individuals within the European Union. Radboud University researchers ranked Llama 2 second-lowest for openness among twenty large language models in July 2023. Mark Dingemanse called the labeling positively misleading due to undocumented training data and poor technical documentation. Ars Technica revised its reporting from open-source to source-available after recognizing restrictions on entities with over 700 million daily active users. The Free Software Foundation classified Llama 3.1 as nonfree software in January 2025. They criticized acceptable use policies that limited popular applications and enforced trade regulations beyond user jurisdictions. A Nature article published later that month argued describing Llama 3 as open amounted to openwashing systems better understood as closed.

  7273747577

• 6. Deployments In Medicine And Space

  ›

  The Stanford University Institute for Human-Centered Artificial Intelligence released Alpaca using the Llama 7B model to acquire capabilities comparable to GPT-3 text-davinci-003. Official files were removed on the 21st of March 2023 due to hosting costs and safety concerns though code remained online. Meditron emerged from École Polytechnique Fédérale de Lausanne and Yale School of Medicine as a family fine-tuned on clinical guidelines and PubMed papers. It demonstrated improved performance on medical benchmarks like MedQA and MedMCQA. Zoom integrated Meta Llama 2 into an AI Companion summarizing meetings and offering presentation tips. Booz Allen Hamilton deployed Llama 3.2 aboard the International Space Station under the Space Llama project. This system ran on Hewlett Packard Enterprise's Spaceborne Computer-2 leveraging NVIDIA CUDA-accelerated computing. Astronauts could retrieve documents via natural language queries without internet connectivity. Reuters reported in 2024 that many Chinese foundation models relied on Llama training data. Researchers from China's People's Liberation Army Academy of Military Sciences developed military tools using the technology despite license prohibitions against such use.

  5356575860