MuZero
On the 19th of November 2019, a team at DeepMind published a preprint introducing MuZero. This moment marked the public debut of an algorithm designed to master games without knowing their rules. The researchers had built a system that could learn dynamics from raw data alone. Previous attempts required explicit programming of game mechanics. This new approach removed that requirement entirely. It represented a shift in how artificial intelligence approached complex decision-making tasks.
MuZero combines tree-based search with learned models to operate without explicit game rules. Unlike its predecessor AlphaZero, this system does not use a simulator that knows the rules. A neural network predicts policy and value for future positions instead. The algorithm learns state transitions through observation rather than code. It handles domains with complex inputs like visual video games effectively. This method allows for more efficient training in classical planning regimes such as Go.
Initial results showed MuZero matched AlphaZero's performance in chess after roughly one million training steps. The system surpassed AlphaZero's performance in Go by one million steps. It also improved on the state of the art in mastering a suite of fifty-seven Atari games. These benchmarks covered visually complex domains known as the Arcade Learning Environment. The algorithm achieved these feats while using fewer computation steps per node in the search tree compared to earlier versions.
The team used sixteen third-generation tensor processing units for training board games. They deployed one thousand TPUs for self-play during those same sessions. Visual environments required eight TPUs for training and thirty-two for self-play. Each step involved five hundred simulations for board games but only fifty for Atari titles. R2D2 had been trained for five days through two million training steps previously. These hardware configurations allowed comparable setups despite improvements in chip power over time.
In late 2021, researchers proposed EfficientZero as a more efficient variant of the original framework. This version achieved 194.3 percent mean human performance with just two hours of real-time game experience. Early 2022 brought Stochastic MuZero to handle chance-based environments like backgammon. That variant uses afterstate dynamics to account for stochastic nature during training. Werner Duvaud produced an open source implementation based on the released pseudocode. The work continues to serve as a reference for generating model-based behavior in other studies.
Up Next
Common questions
When was MuZero first published by DeepMind?
A team at DeepMind published a preprint introducing MuZero on the 19th of November 2019. This event marked the public debut of an algorithm designed to master games without knowing their rules.
How does MuZero differ from AlphaZero in terms of game rules?
MuZero combines tree-based search with learned models to operate without explicit game rules unlike its predecessor AlphaZero. The system learns state transitions through observation rather than code and does not use a simulator that knows the rules.
What performance benchmarks did MuZero achieve against AlphaZero?
Initial results showed MuZero matched AlphaZero's performance in chess after roughly one million training steps. The system surpassed AlphaZero's performance in Go by one million steps and improved on the state of the art in mastering fifty-seven Atari games.
Which hardware configurations were used for training board games versus visual environments?
The team used sixteen third-generation tensor processing units for training board games and deployed one thousand TPUs for self-play during those sessions. Visual environments required eight TPUs for training and thirty-two for self-play.
When was EfficientZero proposed as a variant of the original framework?
In late 2021 researchers proposed EfficientZero as a more efficient variant of the original framework. This version achieved 194.3 percent mean human performance with just two hours of real-time game experience.
All sources
14 references cited across the entry
- 1webDeepMind's MuZero teaches itself how to win at Atari, chess, shogi, and GoKyle Wiggers — VentureBeat — 20 November 2019
- 2newsMuZero figures out chess, rules and allFrederic Friedel — ChessBase GmbH
- 4journalMastering Atari, Go, chess and shogi by planning with a learned modelJulian Schrittwieser et al. — 2020
- 6arxivMastering Chess and Shogi by Self-Play with a General Reinforcement Learning AlgorithmDavid Silver et al. — 5 December 2017
- 7conferenceRECURRENT EXPERIENCE REPLAY IN DISTRIBUTED REINFORCEMENT LEARNINGSteven Kapturowski et al.
- 8journalAN #75: Solving Atari and Go with learned game models, and thoughts from a MIRI employee - LessWrong 2.0Rohin Shah — 27 November 2019
- 11citationwerner-duvaud/muzero-generalWerner Duvaud — 2020-07-15
- 12arxivThe LoCA Regret: A Consistent Metric to Evaluate Model-Based Behavior in Reinforcement LearningHarm van Seijen et al. — 2020-07-06
- 13arxivMastering Atari Games with Limited DataWeirui Ye et al. — 2021-12-11
- 14webPlanning in Stochastic Environments with a Learned ModelIoannis Antonoglou et al. — 2022-01-28