MuZero
On the 19th of November 2019, DeepMind published a preprint describing MuZero, a program that had mastered chess without ever being told the rules. Go, shogi, and 57 Atari video games had fallen to the same approach. The program had access to no rules, no opening books, and no endgame tablebases. The questions that follow from that result are not small. How does a machine win at chess when it has never been told how the pieces move? What does the program actually learn when it plays itself tens of thousands of times? And how had a single algorithm mastered both the ancient geometry of Go and the visually complex world of Atari video games?
DeepMind derived MuZero directly from AlphaZero's code, and the two programs share the same rules for setting hyperparameters. The architectural difference between them, however, is fundamental. AlphaZero relied on a simulator that knew the rules of whatever game it was playing. The simulator tracked which moves were legal, how actions changed the board, and when a search branch should terminate. MuZero replaced that simulator with three separate neural networks, each trained through experience.
The first of MuZero's networks encodes a board state into the program's own internal representation. The second predicts how any given action transforms that representation. The third estimates the policy and value of any given position. AlphaZero accomplished all three tasks with a single unified model.
MuZero ran its search with 20 percent fewer computation steps per node than AlphaZero. Researchers described what the program was doing in plain terms. As they put it: "MuZero really is discovering for itself how to build a model and understand it just from first principles." That internal model remains not fully understood. Whether it is simply encoding game knowledge or hosting additional computation beyond that was identified as an open question for future investigation.
AlphaZero was designed exclusively for two-player games that could end in a win, a draw, or a loss. MuZero dropped that constraint. It operates in single-agent environments with continuous intermediate rewards of arbitrary magnitude and with time discounting applied. That breadth made the algorithm a natural fit for testing in visually complex environments as well.
The 57-game Arcade Learning Environment presented a very different setting from a board game. Each game offers only raw visual pixels as input, with no structured board state for the algorithm to parse. R2D2, the Recurrent Replay Distributed DQN, held the leading position in this benchmark before MuZero's release. R2D2 had been trained over five days and two million training steps.
MuZero surpassed R2D2's mean and median performance across the full suite. Six of the 57 games consistently eluded it, and MuZero never performed well on those. The specific hardware configuration behind those results differed considerably from what AlphaZero had required for its own mastery of board games.
MuZero's board-game training used 16 third-generation tensor processing units for learning, with 1000 more TPUs dedicated to selfplay. The search ran 800 simulations per step. For Atari, the configuration was smaller: 8 TPUs for training and 32 for selfplay, at 50 simulations per step.
AlphaZero had required 64 second-generation TPUs for training and 5000 first-generation TPUs for selfplay. Third-generation chips are individually twice as powerful as second-generation chips, with further advances in bandwidth and networking across chips in a pod. Those improvements made the two training configurations broadly comparable despite the different unit counts.
Against AlphaZero in chess and shogi, MuZero reached matching performance after roughly one million training steps. In Go, it matched AlphaZero at 500,000 steps and surpassed it by one million. Against R2D2 on the Atari suite, MuZero reached the same mean and median scores at 500,000 training steps and moved ahead by one million. Two years after the preprint, a variant called EfficientZero would push those Atari figures significantly further.
Werner Duvaud produced an open-source implementation of MuZero from the pseudocode DeepMind released alongside its initial publication. That work made the approach available to researchers without DeepMind's computing resources. The research community used MuZero as a reference implementation in subsequent projects, including work on generating model-based behavior in new problem settings. The broader view of MuZero's significance cast it as a systems-level achievement more than a pure machine-learning one. It demonstrated that complex, goal-directed behavior could emerge from composing separable, learned components.
In late 2021, a variant called EfficientZero was proposed. It achieved 194.3 percent mean human performance and 109.0 percent median performance on the Atari 100k benchmark using only two hours of real-time game experience. In early 2022, Stochastic MuZero extended the core approach to games involving randomness, such as 2048 and backgammon. It used afterstate dynamics and chance codes to account for unpredictable elements during training. Stochastic MuZero's use of chance codes opened a question about scope. How many other domains, once considered too unpredictable for this class of algorithm, might now fall within reach?
Up Next
Common questions
When was MuZero released and what games did it master?
DeepMind published the MuZero preprint on the 19th of November 2019. The program mastered chess, Go, shogi, and all 57 games in the Arcade Learning Environment through self-play alone, with no access to the rules of any game.
How does MuZero differ from AlphaZero?
MuZero replaces AlphaZero's explicit game-rules simulator with three separate learned neural networks: one for representing board states, one for predicting how actions change those states, and one for estimating policy and value. AlphaZero was restricted to two-player, zero-sum games, while MuZero also works in single-agent environments with continuous rewards of arbitrary magnitude.
What hardware did MuZero use to train on board games and Atari games?
Board-game training used 16 third-generation TPUs for learning and 1000 TPUs for selfplay, running 800 simulations per step. Atari training used 8 TPUs for training and 32 for selfplay, at 50 simulations per step.
How did MuZero perform against R2D2 on the Atari benchmark?
MuZero surpassed R2D2, the Recurrent Replay Distributed DQN, in both mean and median performance across the 57-game Arcade Learning Environment. It never performed well on 6 of those 57 games.
What is EfficientZero and how does it improve on MuZero?
EfficientZero is a variant of MuZero proposed in late 2021. It achieved 194.3 percent mean human performance and 109.0 percent median performance on the Atari 100k benchmark using only two hours of real-time game experience.
What is Stochastic MuZero and how does it differ from MuZero?
Stochastic MuZero is a variant proposed in early 2022 that extends MuZero to games involving randomness, such as 2048 and backgammon. It uses afterstate dynamics and chance codes to account for unpredictable elements when training the dynamics network.
All sources
15 references cited across the entry
- 1DeepMind's MuZero teaches itself how to win at Atari, chess, shogi, and GoKyle Wiggers — VentureBeat — 20 November 2019
- 2NewsMuZero figures out chess, rules and allFrederic Friedel — ChessBase GmbH
- 4JournalMastering Atari, Go, chess and shogi by planning with a learned modelJulian Schrittwieser et al. — 2020
- 6Mastering Chess and Shogi by Self-Play with a General Reinforcement Learning AlgorithmDavid Silver et al. — 5 December 2017
- 7RECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNINGSteven Kapturowski et al.
- 8JournalAN #75: Solving Atari and Go with learned game models, and thoughts from a MIRI employee - LessWrong 2.0Rohin Shah — 27 November 2019
- 11werner-duvaud/muzero-generalWerner Duvaud — 2020-07-15
- 12The LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement LearningHarm van Seijen et al. — 2020-07-06
- 13Mastering Atari Games with Limited DataWeirui Ye et al. — 2021-12-11
- 14Planning in Stochastic Environments with a Learned ModelIoannis Antonoglou et al. — 2022-01-28
- 15MuZero's first step from research into the real worldDeepMind — 11 February 2022