Reinforcement learning
Reinforcement learning is the machine learning paradigm where an agent learns not from labeled data but from the consequences of its own actions. Biological brains treat pain and hunger as negative signals and pleasure and food intake as positive ones. When animals adopt behaviors that optimize those responses, they are, in the formal sense, doing reinforcement learning.
The field stands alongside supervised learning and unsupervised learning as one of the three core approaches in machine learning. Where the other two work with data that is either labeled or unlabeled, reinforcement learning requires neither. It requires only a cycle: the agent acts, the environment shifts, and a reward signal tells the agent how well it did.
Every step of that cycle brings a dilemma. Acting on current knowledge maximizes short-term return but may miss better options that have not yet been tried. Seeking out new possibilities costs immediate reward but may pay off later. That tension, known as the exploration-exploitation dilemma, sits at the center of the field.
From games to autonomous vehicles to the training of systems like ChatGPT, reinforcement learning has moved far beyond the laboratory. A technique called reinforcement learning from human feedback now shapes how some of the most widely used AI systems respond to people. How the mathematics of reward maximization traveled from theory to those applications, and where it still falls short, is what this documentary traces.
In 1997, Burnetas and Katehakis published a rigorous theoretical treatment of the exploration problem. It addressed both the multi-armed bandit setting and finite-state Markov decision processes. Their work established clear foundations where practice had largely outpaced theory.
Epsilon-greedy assigns a parameter, called epsilon, to decide at each step whether to explore or exploit. With probability equal to one minus epsilon, the agent commits to the action it currently rates as best. If multiple actions tie, the choice is made uniformly at random. With probability epsilon, it selects an action entirely at random. Epsilon can be held constant or reduced on a schedule that makes the agent explore progressively less over time. It can also be adjusted adaptively based on heuristics.
Random action selection, with no reference to an estimated probability distribution, consistently performs poorly. For problems with small, finite state spaces, the theoretical behavior of exploration algorithms is well understood. Algorithms that scale to large or infinite state spaces do not yet exist, and simpler methods remain the most practical option for those cases. The theoretical picture of exploration clarifies considerably once the problem is formally modeled as a Markov decision process.
The Markov decision process gives reinforcement learning its mathematical backbone. It specifies a state space, an action space, a transition probability for each state-action pair, and an immediate reward for each transition. Together, these four elements define the environment the agent must navigate.
Full observability means the agent always perceives the exact current state. Partial observability means the agent sees only a subset, or that noise corrupts what it observes. A partially observable problem requires a distinct formulation, the partially observable Markov decision process. Actions can also be constrained by state. If the current value of an account balance is 3 and a transition would subtract 4, that action is simply blocked.
When the agent's performance falls short of a hypothetical optimal agent, the gap is called regret. Acting near-optimally requires reasoning about long-term consequences, even when the immediate reward for doing so is negative. A discount factor less than 1 ensures that rewards in the distant future count for less than rewards in the immediate future.
AlphaGo represents the framework applied to the game of Go. The same structure underlies applications in backgammon, checkers, robot control, energy storage, photovoltaic generators, and autonomous driving systems.
A policy is considered optimal only if it achieves the best expected return from any initial state. This is a stronger requirement than doing well from a typical starting point. Meeting it without testing every possible policy by brute force is the problem that reinforcement learning algorithms are designed to solve.
Brute force is the naive approach: sample returns for each possible policy, then select whichever yields the highest expected discounted return. Two obstacles make this impractical. The number of distinct policies can be large or even infinite. The variance of returns is often high, requiring many samples to form a reliable estimate.
Monte Carlo methods sidestep this by averaging sample returns gathered from actual or simulated experience. They apply specifically to episodic tasks, where experience divides into episodes that each eventually terminate. Value function and policy updates happen only after a complete episode finishes. Crucially, only a model capable of generating sample transitions is needed; the full specification of transition probabilities required by dynamic programming is not.
Sutton's temporal difference methods address the high-variance problem that can afflict Monte Carlo approaches. TD methods ground their updates in the recursive Bellman equation and can revise estimates after each individual transition, rather than waiting for an episode to end. Batch variants, such as the least-squares temporal difference method, process stored transitions together and can use each sample more efficiently. A tunable parameter allows continuous interpolation between pure Monte Carlo averaging and the basic temporal difference update.
Williams's REINFORCE method searches directly in the space of policies rather than estimating values first. Known in simulation-based optimization as the likelihood ratio method, it constructs gradient estimates from sampled trajectories. Gradient-free alternatives include simulated annealing, cross-entropy search, and methods from evolutionary computation. These can reach a global optimum in theory, though they may converge slowly when data is noisy or episode trajectories are long.
The Dyna algorithm bridges learning and planning by learning a model of the Markov decision process from experience. It then uses that model to generate additional simulated transitions, supplementing real experience to speed up learning. Actor-critic methods merge value-function estimation with direct policy search. They emerged from these lines of work and became central to the next generation of reinforcement learning, built on deep neural networks.
Google DeepMind's work on learning to play Atari games drew widespread attention to deep reinforcement learning. In this approach, also called end-to-end reinforcement learning, a deep neural network serves as the function approximator. The state space does not need to be designed by hand.
The Deep Q-Network, or DQN, is an off-policy algorithm built for environments with discrete action spaces and continuous state spaces. For continuous control, Deep Deterministic Policy Gradient (DDPG) and Soft Actor-Critic (SAC) operate off-policy. The Asynchronous Advantage Actor-Critic algorithm, A3C, is on-policy and uses an advantage function, defined as action-value minus state-value. Proximal Policy Optimization (PPO) and Trust Region Policy Optimization (TRPO) are on-policy methods that handle both discrete and continuous action spaces. Twin Delayed Deep Deterministic Policy Gradient, or TD3, is an off-policy algorithm for continuous control. The Distributional Soft Actor Critic, DSAC, models the full distribution of action values rather than their expectation alone.
Adversarial deep reinforcement learning has emerged as a focused research area because learned policies are susceptible to imperceptible manipulations. Proposed defenses have been studied, but recent work suggests they fall well short of accurately representing the true scope of current vulnerabilities. The same neural approximation techniques that made deep RL successful in games would later find their way into the training of language models.
Since the early 2020s, reinforcement learning has become central to natural language processing. Many NLP tasks involve sequential decision-making where quality depends on optimizing long-term goals rather than predicting a single correct label. Dialogue generation, text summarization, and machine translation all share this structure.
Policy gradient and sequence-level training were among the earliest reinforcement learning tools applied to dialogue systems. Conversations were treated as sequences of actions to be optimized for fluency and coherence. These early techniques laid a foundation for a much larger shift.
Reinforcement learning from human feedback, or RLHF, replaces the hand-coded reward signal with one derived from human judgment. Human raters evaluate model outputs, their assessments train a separate reward model, and that reward model then guides the reinforcement learning agent. Unlike rule-based or fully supervised systems, RLHF can align model behavior with human judgment on tasks that are complex and subjective.
InstructGPT was an early application of this approach: a language model trained specifically to follow human instructions. ChatGPT incorporated RLHF to improve response quality and to address safety concerns.
Offline reinforcement learning has extended these ideas to dialogue without requiring live human interaction during training. Such methods optimize for engagement, coherence, and diversity by drawing on past conversation logs combined with pre-trained reward models.
DeepSeek-R1 represents a departure from the RLHF template. It incorporates multi-stage training and cold-start data before the reinforcement learning phase begins. The model achieves performance comparable to OpenAI-o1-1217 on reasoning tasks. It was trained via large-scale reinforcement learning without supervised fine-tuning as a preliminary step. That result suggests extensive human-labeled data may not always be a prerequisite for strong performance.
OpenAI's Dota-playing bot required the equivalent of thousands of years of simulated gameplay before reaching human-level performance. That figure captures reinforcement learning's most persistent obstacle: extreme sample inefficiency. Techniques such as experience replay and curriculum learning have been proposed to reduce the number of interactions required. They add complexity and do not always succeed in real-world settings.
A small change in a policy or in the environment can trigger extreme fluctuations in performance. This instability is particularly acute in continuous or high-dimensional action spaces, where the learning process becomes less predictable and harder to control.
Agents trained in specific environments often fail to transfer their learned behaviors to new or unfamiliar scenarios. The ability to adapt without extensive retraining is precisely what many real-world deployments require, yet it remains difficult to build into current systems.
Poorly specified reward functions can lead agents to develop unintended behaviors that satisfy the letter of the reward while violating the intent. Training on biased data can compound this: policies may perpetuate or amplify existing biases, producing discriminatory or otherwise undesirable outcomes.
Safe reinforcement learning defines the problem as learning policies that maximize return while respecting safety constraints throughout both training and deployment. A risk-averse variant targets the conditional value at risk, or CVaR, rather than expected return. CVaR also increases robustness to model uncertainties, though it requires special care to prevent gradient bias and what practitioners call blindness to success.
Self-reinforcement learning, introduced in 1982, takes a different architectural path entirely. It replaces the external reward signal with an internal mechanism built around feelings and emotions. The Crossbar Adaptive Array, or CAA, is the neural network introduced alongside this paradigm. It computes action decisions and emotional responses simultaneously in a crossbar structure, with initial conditions drawn from the genetic environment. It accepts one input, the current situation, and produces one output, the action to be taken. The system is driven by the interaction between cognition and emotion.
Active research connects temporal difference learning to the neuroscience of dopamine. Dopaminergic projections from the substantia nigra to the basal ganglia appear to function as a biological prediction error signal. If that analogy holds, the mathematics developed to train machines may ultimately describe the same mechanism the brain uses to learn.
Up next
Updated
Common questions
What is reinforcement learning and how does it work?
Reinforcement learning is a machine learning paradigm in which an agent learns by interacting with an environment and receiving reward signals. The agent acts, the environment moves to a new state, and a reward is issued for the transition. The agent's goal is to find a policy that maximizes cumulative reward over time.
What is the exploration-exploitation dilemma in reinforcement learning?
The exploration-exploitation dilemma in reinforcement learning is the tension between trying new actions to discover better outcomes and using current knowledge to take the best known action. Burnetas and Katehakis (1997) provided a rigorous theoretical treatment of this problem. The epsilon-greedy method addresses it by assigning a probability, epsilon, to random exploration and the complementary probability to exploiting the current best action.
What is reinforcement learning from human feedback and which AI models use it?
Reinforcement learning from human feedback, or RLHF, trains a reward model from human evaluator ratings and then uses that model to guide the reinforcement learning agent. InstructGPT was an early language model trained this way, and ChatGPT incorporated RLHF to improve response quality and address safety concerns.
What are the main deep reinforcement learning algorithms?
Deep reinforcement learning algorithms use neural networks as function approximators instead of hand-designed state representations. DQN handles discrete action spaces off-policy; DDPG and SAC handle continuous control off-policy; A3C is an on-policy actor-critic method that uses an advantage function; PPO and TRPO are on-policy methods covering both discrete and continuous actions; TD3 and DSAC are additional off-policy continuous-control algorithms.
What are the main limitations and challenges of reinforcement learning?
Reinforcement learning faces three major challenges: sample inefficiency, training instability, and poor generalization to new environments. OpenAI's Dota-playing bot required thousands of years of simulated gameplay to reach human-level performance, illustrating the cost of sample inefficiency. Poorly designed reward functions can also lead agents to develop unintended behaviors that satisfy the reward's letter while violating its intent.
What is DeepSeek-R1 and how does it use reinforcement learning?
DeepSeek-R1 is a language model trained via large-scale reinforcement learning without supervised fine-tuning as a preliminary step. It uses multi-stage training and cold-start data before the reinforcement learning phase begins. The model achieves performance comparable to OpenAI-o1-1217 on reasoning tasks.
All sources
88 references cited across the entry
- 2JournalReinforcement Learning: A SurveyLeslie P. Kaelbling et al. — 1996
- 3BookReinforcement Learningvan Otterlo, M. et al. — 2012
- 4BookReinforcement Learning for Sequential Decision and Optimal ControlShengbo Li — 2023
- 5BookArtificial intelligence: a modern approachStuart J. Russell et al. — Prentice Hall — 2010
- 6JournalNeural Basis of Reinforcement Learning and Decision MakingDaeyeol Lee et al. — 21 July 2012
- 7JournalCommunity energy storage operation via reinforcement learning with eligibility tracesEdgar Mauricio Salazar Duque et al. — 2022
- 8ALLSTEPS: Curriculum-driven Learning of Stepping Stone SkillsZhaoming Xie — 2020
- 9JournalOptimal dispatch of PV inverters in unbalanced distribution systems using Reinforcement LearningPedro P. Vergara et al. — 2022
- 10JournalSelf-Learned Intelligence for Integrated Decision and Control of Automated Vehicles at Signalized IntersectionsYangang Ren et al. — 2026
- 11BookSimulation-based Optimization: Parametric Optimization Techniques and ReinforcementAbhijit Gosavi — Springer — 2003
- 12Optimal adaptive policies for Markov Decision ProcessesApostolos N. Burnetas et al. — 1997
- 13KI 2011: Advances in Artificial IntelligenceMichel Tokic et al. — Springer — 2011
- 15JournalReinforcement learning with replacing eligibility tracesSatinder P. Singh et al. — 1996-03-01
- 16ThesisTemporal Credit Assignment in Reinforcement LearningRichard S. Sutton — University of Massachusetts, Amherst, MA — 1984
- 17JournalLearning to predict by the method of temporal differencesSteven J. Bradtke et al. — 1996
- 18ThesisLearning from Delayed RewardsChristopher J.C.H. Watkins — King's College, Cambridge, UK — 1989
- 19JournalDetection of Static and Mobile Targets by an Autonomous Agent with Deep Q-Learning AbilitiesBarouch Matzliach et al. — 2022
- 20A class of gradient-estimating algorithms for reinforcement learning in neural networksRonald J. Williams — 1987
- 21Reinforcement Learning for Humanoid RoboticsJan Peters et al. — 2003
- 22Simple Reinforcement Learning with Tensorflow Part 8: Asynchronous Actor-Critic Agents (A3C)Arthur Juliani — 2016-12-17
- 23BookA Survey on Policy Search for RoboticsMarc Peter Deisenroth et al. — NOW Publishers — 2013
- 24Integrated Architectures for Learning, Planning and Reacting based on Dynamic ProgrammingRichard Sutton — 1990
- 26Meta-LearningLan Zou — Academic Press — 2023-01-01
- 27When to use parametric models in reinforcement learning?Hado van Hasselt et al. — 2019
- 29JournalEfficient Model Learning Methods for Actor–Critic ControlIvo Grondman et al. — 2012-06-01
- 31Reinforcement Learning with Temporal Logic RewardsXiao Li et al. — 2017
- 32JournalReward Machines: Exploiting Reward Function Structure in Reinforcement LearningRodrigo Toro Icarte et al. — 2022
- 33JournalA probabilistic argumentation framework for reinforcement learning agentsRégis Riveret et al. — 2019
- 34Entity-Centric Reinforcement Learning for Object Manipulation from PixelsHaramati, Dan et al. — 2024
- 35Entity-based Reinforcement Learning for Autonomous Cyber DefenceIsaac Symes Thompson et al. — ACM — 2024-11-07
- 36Entity-Based Reinforcement LearningClemens Winter — 2023-04-14
- 37Reinforcement Learning with Feedback from Multiple Humans with Diverse SkillsTaku Yamagata et al. — 2021-11-16
- 38JournalHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic MotivationTejas D. Kulkarni et al. — Curran Associates Inc. — 2016
- 40Book2020 Design, Automation & Test in Europe Conference & Exhibition (DATE)Somdip Dey et al. — March 2020
- 41Smartphones get smarter with Essex innovationTony Quested
- 42Future smartphones 'will prolong their own battery life by monitoring owners' behaviour'Rhiannon Williams — 2020-07-21
- 43BookEmbodied Artificial IntelligenceF. Kaplan et al. — Springer — 2004
- 44JournalKeep your options open: an information-based driving principle for sensorimotor systemsA. Klyubin et al. — 2008
- 45BookIntrinsically Motivated Learning in Natural and Artificial SystemsA. G. Barto — Springer — 2013
- 46JournalDeep Execution - Value and Policy Based Reinforcement Learning for Trading and Beating Market BenchmarksKevin Dabérius et al. — 2020
- 47JournalSelf-organizing maps for storage and transfer of knowledge in reinforcement learningThommen George Karimpanal et al. — 2019
- 48Sutton, Barto (2018) p. Section 5.4, p. 100Sutton, Barto — 2018
- 49JournalAsynchronous Methods for Deep Reinforcement LearningVolodymyr Mnih et al. — PMLR — 16 June 2016
- 50JournalDistributional Soft Actor-Critic: Off-policy reinforcement learning for addressing value estimation errorsJ Duan et al. — 2021
- 51Book2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)Y Ren et al. — 2020
- 52JournalDistributional Soft Actor-Critic with Three RefinementsJ Duan et al. — 2025
- 53BookDynamic, Genetic and Chaotic Programming: The Sixth-Generation Computer Technology SeriesBranko Soucek — John Wiley & Sons, Inc — 6 May 1992
- 54JournalAn Introduction to Deep Reinforcement LearningVincent Francois-Lavet — 2018
- 55JournalHuman-level control through deep reinforcement learningVolodymyr Mnih — 2015
- 56JournalExplaining and Harnessing Adversarial ExamplesIan Goodfellow et al. — 2015
- 57BookMachine Learning and Data Mining in Pattern RecognitionVahid Behzadan et al. — 2017
- 58BookAdversarial Attacks on Neural Network PoliciesSandy Huang et al. — 2017-02-07
- 59JournalDeep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs.Ezgi Korkmaz — 2022
- 60BookProceedings of 1994 IEEE 3rd International Fuzzy Systems ConferenceH.R. Berenji — IEEE — 1994
- 61Book2017 IEEE 15th International Symposium on Applied Machine Intelligence and Informatics (SAMI)David Vincze — IEEE — 2017
- 62BookProceeding ICML '00 Proceedings of the Seventeenth International Conference on Machine LearningA. Y. Ng et al. — Morgan Kaufmann Publishers — 2000
- 63JournalMaximum entropy inverse reinforcement learningBrian D. Ziebart et al. — AAAI Press — 2008-07-13
- 64JournalTrajectory modeling via random utility inverse reinforcement learningAnselmo R. Pitombeira-Neto et al. — March 2024
- 65JournalA practical guide to multi-objective reinforcement learning and planningHayes C, Radulescu R, Bargiacchi E, et al — 2022
- 66BookMultiple Attribute Decision Making: Methods and ApplicationsGwo-Hshiung Tzeng et al. — CRC Press — 2011
- 67JournalA review of safe reinforcement learning: Methods, theories and applicationsShangding Gu et al. — 10 September 2024
- 68JournalA comprehensive survey on safe reinforcement learningJavier García et al. — 1 January 2015
- 69JournalImplicit Quantile Networks for Distributional Reinforcement LearningWill Dabney et al. — PMLR — 2018-07-03
- 70JournalRisk-Sensitive and Robust Decision-Making: a CVaR Optimization ApproachYinlam Chow et al. — Curran Associates, Inc. — 2015
- 72JournalOptimizing the CVaR via SamplingAviv Tamar et al. — 2015-02-21
- 73JournalEfficient Risk-Averse Reinforcement LearningIdo Greenberg et al. — 2022-12-06
- 75JournalImplementation Matters in Deep RL: A Case Study on PPO and TRPOLogan Engstrom et al. — 2019-09-25
- 76JournalDistributional Soft Actor-Critic with Three RefinementsCédric Colas — 2019-03-06
- 77JournalReward Machines: Exploiting Reward Function Structure in Reinforcement LearningIdo Greenberg et al. — PMLR — 2021-07-01
- 78JournalSurvey on reinforcement learning for language processingVíctor Uc-Cetina et al. — 2022
- 79Sequence Level Training with Recurrent Neural NetworksMarc'Aurelio Ranzato et al. — 2015
- 80A Deep Reinforced Model for Abstractive SummarizationRomain Paulus et al. — 2017
- 81Deep Reinforcement Learning for Dialogue GenerationJiwei Li et al. — 2016
- 82Deep Reinforcement Learning from Human PreferencesPaul F. Christiano et al. — 2017
- 83Learning to Summarize from Human FeedbackNisan Stiennon et al. — 2020
- 84Training Language Models to Follow Instructions with Human FeedbackLong Ouyang et al. — 2022
- 85Direct Preference Optimization: Your Language Model is Secretly a Reward ModelRafael Rafailov et al. — 2023
- 86Tülu 3: Pushing Frontiers in Open Language Model Post-TrainingNathan Lambert et al. — 2024
- 87OpenAI o1 System CardOpenAI — 2024
- 88JournalDeepSeek-R1 incentivizes reasoning in LLMS through reinforcement learningDeepSeek-AI — January 22, 2025