DeepSeek

• 1. Introduction

  ›

  DeepSeek, a Chinese artificial intelligence company based in Hangzhou, helped trigger the largest single-company loss of value in the history of the US stock market. Nvidia, an established leader in AI hardware, saw its share price drop sharply and US$600 billion of its market value disappear. The cause was not a rival chipmaker. It was a chatbot and a family of language models from a company that a hedge fund owns and funds.

  That raises a string of questions. How does a hedge fund end up building language models? How did its engineers keep pace with far larger rivals while American export rules limited the chips available to Chinese buyers? What exactly did the company share with the world, and what did it hold back? And why did a few weeks of software releases shake markets on another continent?

  14441716

• 2. From Trading Desk to Lab

  ›

  In February 2016, an AI enthusiast named Liang Wenfeng co-founded a hedge fund called High-Flyer. Liang had been trading since the 2008 financial crisis, while he was still a student at Zhejiang University.

  On the 21st of October 2016, the firm began trading stocks with a deep learning model that depended on GPUs. Until then it had relied on linear models running on CPUs. By the end of 2017, most of its trading was driven by AI. By 2021, the firm used AI exclusively, often on Nvidia chips.

  On the 14th of April 2023, High-Flyer announced a research lab for artificial general intelligence. The lab's stated focus was AI tools unrelated to the firm's financial business. Two months later, on the 17th of July 2023, it was spun off as an independent company named DeepSeek, with High-Flyer as its principal investor and backer.

  Venture capital firms had been reluctant to fund the venture. They considered it unlikely to produce a quick exit. Without outside money, DeepSeek's future would rest on its parent's cash and on the machines that parent had already bought.

  819223620

• 3. Fire-Flyer

  ›

  In 2019, High-Flyer began building its first computing cluster, called Fire-Flyer, at a cost of 200 million yuan. It held 1,100 GPUs linked at 200 Gbit/s. It was retired after only 1.5 years in operation.

  By 2021, Liang was buying Nvidia GPUs in large quantities for an AI project. He reportedly obtained 10,000 A100 chips before the United States restricted chip sales to China. That same year, construction began on Fire-Flyer 2, with a budget of 1 billion yuan.

  In 2022, Fire-Flyer 2 was reportedly running at over 96% of its capacity, totaling 56.74 million GPU hours. About 27% of that time went to scientific computing outside the company.

  The cluster that year held 5,000 PCIe A100 GPUs, spread across 625 nodes of eight GPUs each. Choosing PCIe over the DGX version of the A100 was deliberate. The models then in training fit inside a single 40 GB GPU memory, so they needed data parallelism but not model parallelism. Later, engineers added NVLinks and the Nvidia Collective Communications Library to train larger models. The cluster is split into two zones, and its network uses two fat trees, a layout chosen for high bisection bandwidth. As of 2025, it is still in operation.

  The software around the cluster has five named parts. 3FS, the Fire-Flyer File System, is a distributed file system built for asynchronous random reads. It uses Direct I/O, which skips caching, because each piece of data is read once at random and never reused. HaiScale Distributed Data Parallel implements several forms of parallelism, including pipeline, tensor and experts parallelism, plus FSDP and ZeRO. The HAI Platform handles task scheduling, fault handling and disaster recovery. A library called hfai.nn supplies common neural network operators, much like torch.nn in PyTorch.

  The fifth part, hfreduce, was originally meant to replace the Nvidia library for communication. It combines gradients during backpropagation, and it runs asynchronously on the CPU so that GPU kernels never wait on it. Like the Nvidia library, it uses a two-tree broadcast.

  3738394036

• 4. Efficiency by Design

  ›

  DeepSeek Coder arrived on the 2nd of November 2023, and the DeepSeek-LLM series followed on the 29th of November. Coder was a set of eight models, four pretrained and four instruction-finetuned, all with 16K context lengths. Its pretraining data totaled 1.8T tokens, of which 87% was source code. A license restricted downstream use to what it called "open and responsible downstream usage".

  The LLM series came in 7B and 67B sizes. It trained on 2 trillion tokens of English and Chinese text drawn from a deduplicated Common Crawl. Its architecture was essentially that of Meta's Llama series. The accompanying paper claimed results above Llama 2 and most open-source models of the time.

  In January 2024, the company released two DeepSeek-MoE models with 16B parameters each, of which 2.7B were activated per token. Mixture of experts splits a network's feedforward layers into many modules and activates only the highest-scoring ones. The trouble is that some experts get overused while others sit idle. Forcing balance makes experts copy each other's skills. DeepSeek's answer was "shared experts" that are always queried and learn core skills, leaving "routed experts" to learn rarer, peripheral ones. In April, three DeepSeek-Math models followed. Their final stage used Group Relative Policy Optimization, a variant of Proximal Policy Optimization developed by the team, with a dataset of 144K math questions.

  In May 2024, DeepSeek-V2 changed the architecture. It had 236B parameters but only 21B active, and it trained on 8.1T tokens, with 12% more Chinese than English. Multi-head latent attention replaced standard attention. It compresses latent vectors to shrink the key-value cache and so cut memory use. The Financial Times reported a price of 2 RMB for every million output tokens, cheaper than its peers. The University of Waterloo Tiger Lab ranked V2 seventh on its leaderboard.

  DeepSeek-V3 in December 2024 kept the V2 design and added multi-token prediction, which decodes extra tokens faster but less accurately. It had 671B parameters with 37B active, and it trained on 14.8T tokens. The team did low-level engineering to squeeze the hardware. Much of the forward pass ran in 8-bit floating point numbers instead of the standard 32-bit. On each H800 GPU, 20 of 132 streaming multiprocessors did nothing but inter-GPU communication. Every 10 minutes, the system rearranged which machine hosted each expert to even out the load. Benchmarks showed V3 outperforming Llama 3.1 and Qwen 2.5 while matching GPT-4o and Claude 3.5 Sonnet.

  606162672

• 5. Teaching Models to Reason

  ›

  On the 20th of November 2024, a preview called DeepSeek-R1-Lite became available through chat. It was trained for logical inference, mathematical reasoning and real-time problem-solving. DeepSeek claimed it beat OpenAI's o1 on benchmarks such as the American Invitational Mathematics Examination and MATH. The Wall Street Journal reported otherwise for one test. On 15 problems from the 2024 exam, o1 reached a solution faster.

  The model that mattered most was DeepSeek-R1-Zero. It started from DeepSeek-V3-Base and was trained purely with reinforcement learning, with no supervised finetuning at all. Every reward was rule-based, mainly of two kinds. An accuracy reward checked whether a boxed math answer was correct or whether code passed its tests. A format reward checked that the model put its thinking inside a designated tag.

  R1-Zero had a flaw. Its answers were hard to read, and it mixed languages. To fix this, the team first finetuned the base model on "thousands" of cold-start examples built for readability. They then repeated the reinforcement learning with an added "language consistency reward" that pushed the model to answer in one language.

  That produced an internal model that was never released. From it, the team synthesized 600K reasoning samples, discarding any with a wrong final answer. They added 200K non-reasoning samples covering writing, factual question answering, self-cognition and translation. DeepSeek-V3-Base was finetuned on the resulting 800K samples for 2 epochs. A final round of reinforcement learning, mixing rule-based and model-based rewards, produced DeepSeek-R1.

  Smaller distilled models took a different route. They started from open-weight models such as Llama and Qwen and were finetuned on 800K samples synthesized from R1. They received no reinforcement learning. The V3 training had earlier avoided R1 itself as a teacher, since its output suffered "overthinking, poor formatting, and excessive length".

  With the January 2025 release, the team posted a preprint on arXiv. An updated version appeared in Nature in September 2025.

  67104813

• 6. The January Shock

  ›

  On the 20th of January 2025, DeepSeek launched a free chatbot built on R1 for iOS and Android. By the 27th of January, it had passed ChatGPT as the most downloaded freeware app on the iOS App Store in the United States. Nvidia's share price fell 18%.

  Part of the alarm was about money. The company said it trained V3 for US$6 million, against US$100 million for OpenAI's GPT-4 in 2023. It also claimed about one-tenth of the computing power Meta used for Llama 3.1. The published breakdown listed 2,788 thousand GPU hours at US$5.576 million, mostly pre-training at 2,664 thousand hours. Context extension took 119 thousand hours and fine-tuning only 5 thousand. Critics called the figure misleading, because it covers only part of the true cost.

  The other part was how the work was done. DeepSeek trained during trade restrictions on AI chip exports to China, using weaker chips intended for export and fewer of them. Techniques such as mixture of experts layers cut the expense further. R1 gave responses comparable to GPT-4o and o1. Observers described the result as "upending AI".

  Investors and pundits reached for a historical comparison, the "Sputnik moment" for the United States in AI. It hit hardest because the models were open-source, cheap and strong at once.

  The price effects spread inside China. Some investors expected a price war in the American AI industry, and R1 was dubbed the "Pinduoduo of AI". ByteDance, Tencent, Baidu and Alibaba all cut the price of their own AI models. Despite its low prices, DeepSeek was profitable, unlike rivals that were losing money.

  4344141657

• 7. A Lab, Not a Shop

  ›

  As of May 2024, Liang personally held an 84% stake in DeepSeek through two shell corporations. He also serves as chief executive of both DeepSeek and High-Flyer.

  DeepSeek has said it focuses on research and has no immediate plans for commercialization. That stance also lets it skirt certain provisions of China's AI regulations aimed at consumer-facing technologies.

  Its hiring favors skills over lengthy work experience, so many recruits come straight from university. The company also hires people outside computer science to widen what its models know, with poetry and advanced mathematics as examples. According to The New York Times, dozens of researchers have, or once had, affiliations with People's Liberation Army laboratories and the Seven Sons of National Defence.

  The company describes its models as "open weight". The exact parameters are public, but the training data is not openly licensed, which gives less freedom to modify than true open source. Since the debut of R1, new models have mostly carried the MIT License, a shift from the restricted DeepSeek License on the earliest ones.

  US chip restrictions pushed the engineers to refine their algorithms for maximum efficiency. That let them work with older hardware and use less energy. The same economy found a market in Africa, where DeepSeek offers cheaper, less power-hungry AI. The company has strengthened African language models and helped spawn startups, including some in Nairobi. Working with Huawei's storage and cloud services, it offers local data sovereignty and more flexibility than Western platforms.

  82323413

• 8. After R1

  ›

  On the 24th of March 2025, DeepSeek released DeepSeek-V3-0324 under the MIT License. R1's planned successor, R2, was reportedly due in early May 2025. Instead, on the 28th of May, R1 was updated to R1-0528. This version has been noted for following official Chinese Communist Party ideology and censorship more tightly than earlier models.

  In early July, R2 had still not appeared, because Liang was not yet satisfied with its performance. It was still missing in August. Sources cited slow data labelling and chip problems. Authorities had encouraged DeepSeek to train on Huawei's Ascend chips, but those had stability issues, slower connections between chips and inferior software. DeepSeek went back to Nvidia chips for training and kept Huawei chips for inference. Separately, the Cyberspace Administration of China reportedly asked several large corporations to stop buying Nvidia's H20 and buy from domestic suppliers.

  On the 21st of August 2025, DeepSeek V3.1 added a hybrid design with thinking and non-thinking modes. It beat earlier V3 and R1 results by over 40% on certain benchmarks, such as SWE-bench and Terminal-bench. V3.1-Terminus followed on the 22nd of September, and V3.2-Exp on the 29th of September. The latter used DeepSeek Sparse Attention, an efficiency method built on research published in February. DeepSeek-V3.2 arrived on the 1st of December 2025, with a Speciale variant focused on reasoning.

  In February 2026, Anthropic accused DeepSeek of using thousands of fraudulent accounts to generate millions of conversations with Claude, to train its own models. In April 2026, investors began talks about a $300 million funding round that would value the company at $10 billion.

  On the 24th of April 2026, DeepSeek previewed its V4 series under the MIT License. DeepSeek-V4-Pro has 1.6 trillion parameters and DeepSeek-V4-Flash has 284 billion. Both have a 1-million token context window. Huawei and Cambricon, a chipmaker, have adopted V4.

  2526525455