Reinforcement learning
In 1956, Arthur Samuel wrote about machine learning that could improve through experience rather than explicit programming. This early concept laid the groundwork for what would become reinforcement learning decades later. The field defines an agent as any entity that takes actions within a dynamic environment to maximize a reward signal. Unlike supervised learning which relies on labeled data or unsupervised learning which finds patterns in unlabeled data, reinforcement learning trains agents through direct interaction with their surroundings.
The environment is typically modeled as a Markov decision process where states transition based on actions taken by the agent. At each discrete time step, the agent receives the current state and reward information before choosing its next action from available options. The environment then moves to a new state while determining the associated reward for that specific transition. This cycle continues until the agent learns a policy that maximizes expected cumulative rewards over time.
Biological brains appear hardwired to interpret signals like pain and hunger as negative reinforcements while treating pleasure and food intake as positive ones. Animals learn behaviors that optimize these rewards, suggesting natural reinforcement learning mechanisms exist in nature. The mathematical framework assumes full observability when agents directly observe environmental states, though partial observability occurs when only subsets of states are accessible or corrupted by noise.
The trade-off between exploration and exploitation has been most thoroughly studied through the multi-armed bandit problem and finite state space Markov decision processes in Burnetas and Katehakis (1997). Reinforcement learning requires clever exploration mechanisms because randomly selecting actions without reference to estimated probability distributions shows poor performance overall. With small finite Markov decision processes, researchers understand the dynamics relatively well compared to problems with infinite state spaces.
One practical method is epsilon-greedy where epsilon serves as a parameter controlling how much exploration versus exploitation occurs. With probability one minus epsilon, exploitation gets chosen meaning the agent selects the action it believes has the best long-term effect. When ties occur between actions, they break uniformly at random. Alternatively, with probability epsilon, exploration happens and the action gets chosen uniformly at random from all possibilities.
Epsilon usually remains a fixed parameter but can be adjusted according to schedules making agents explore progressively less over time. Adaptive adjustments based on heuristics also allow flexibility depending on specific application requirements. Simple exploration methods remain the most practical choice due to lack of algorithms that scale well with increasing numbers of states or infinite state spaces.
Monte Carlo methods solve reinforcement learning problems by averaging sample returns rather than requiring full knowledge of environment dynamics. These approaches rely solely on actual or simulated sequences of states, actions, and rewards obtained through interaction with an environment. Learning from real experience does not require prior environmental knowledge yet still leads to optimal behavior when conditions permit.
Temporal difference methods address issues found in Monte Carlo approaches by allowing procedures to change policies before values settle completely. Sutton's temporal difference techniques use recursive Bellman equations to compute expectations incrementally after each transition while discarding old transitions immediately. Batch methods like least-squares temporal difference utilize information in samples more effectively though incremental methods become necessary when computational complexity prevents batch processing.
Q-learning emerged as a value iteration starting point leading to many variants including Deep Q-learning when neural networks represent Q functions for stochastic search problems. Actor-critic methods combine policy search with value function estimation performing well across various problem domains since their inception. The Dyna algorithm learns models from experience using them to provide modeled transitions alongside real transitions for enhanced value function updates.
Reinforcement learning has been applied successfully to energy storage systems robot control photovoltaic generators backgammon checkers Go AlphaGo and autonomous driving systems. OpenAI's Dota-playing bot utilized thousands of years of simulated gameplay to achieve human-level performance despite requiring massive computational resources. Techniques like experience replay and curriculum learning attempt to reduce sample inefficiency but add complexity that sometimes proves insufficient for real-world applications.
Google DeepMind increased attention to deep reinforcement learning through work on learning ATARI games without explicitly designing state spaces. This approach extends reinforcement learning by using deep neural networks enabling end-to-end learning capabilities previously unavailable. Policy gradient and sequence-level training techniques laid foundations for broader application to dialogue generation text summarization machine translation and other sequential decision-making tasks.
In robotics contexts, policy search methods get stuck in local optima frequently because they rely on local search strategies. Continuous or high-dimensional action spaces make learning steps more complex and less predictable compared to discrete environments. Despite these challenges, successful implementations demonstrate the field's potential for physical control systems and autonomous navigation scenarios.
A major breakthrough happened with introduction of reinforcement learning from human feedback where human feedback ratings train reward models guiding RL agents. Unlike traditional rule-based or supervised systems this method allows models to align behavior with human judgments on complex subjective tasks. The technique initially appeared in development of InstructGPT an effective language model trained to follow human instructions before appearing later in ChatGPT which incorporates RLHF for improving output responses ensuring safety measures.
DeepSeek-R1 achieves performance comparable to OpenAI-o1-1217 on reasoning tasks through large-scale reinforcement learning without supervised fine-tuning as preliminary step. Researchers explore offline reinforcement learning in natural language processing to improve dialogue systems without needing live human interaction continuously. These methods optimize user engagement coherence diversity based on past conversation logs and pre-trained reward models available for analysis.
Early applications emerged in dialogue systems where conversations determined series actions optimized for fluency and coherence overall. Policy gradient techniques combined with sequence-level training established foundations enabling broader application across multiple areas of natural language processing today. Human-centered goals replace single correct label predictions when quality depends on optimizing long-term outcomes instead.
Training deep neural network-based models creates instability prone to divergence from small changes in policies or environments causing extreme fluctuations in performance levels. Continuous or high-dimensional action spaces make learning steps more complex less predictable compared to simpler discrete environments. Generalization struggles occur when agents trained in specific environments fail to apply learned policies to new unseen scenarios effectively.
Designing appropriate reward functions critical because poorly designed structures lead unintended behaviors perpetuating existing biases through discriminatory unfair outcomes. RL systems trained on biased data may reinforce discrimination while failing to achieve desired behavioral objectives consistently. Developing algorithms transferring knowledge across tasks without extensive retraining represents major setback preventing dynamic real-world environment applications where adaptability crucially matters.
Up Next
Common questions
What is reinforcement learning and how does it differ from supervised learning?
Reinforcement learning defines an agent as any entity that takes actions within a dynamic environment to maximize a reward signal. Unlike supervised learning which relies on labeled data, this field trains agents through direct interaction with their surroundings.
When did Arthur Samuel write about machine learning that could improve through experience?
Arthur Samuel wrote about machine learning that could improve through experience rather than explicit programming in 1956. This early concept laid the groundwork for what would become reinforcement learning decades later.
How do epsilon-greedy methods control exploration versus exploitation in reinforcement learning?
Epsilon serves as a parameter controlling how much exploration versus exploitation occurs where one minus epsilon probability selects the best action. When ties occur between actions they break uniformly at random while epsilon probability chooses an action uniformly at random from all possibilities.
Which organizations developed AlphaGo and ChatGPT using reinforcement learning techniques?
Google DeepMind increased attention to deep reinforcement learning through work on learning ATARI games without explicitly designing state spaces. The technique initially appeared in development of InstructGPT before appearing later in ChatGPT which incorporates RLHF for improving output responses ensuring safety measures.
What challenges arise when applying reinforcement learning to continuous or high-dimensional action spaces?
Continuous or high-dimensional action spaces make learning steps more complex less predictable compared to discrete environments. Policy search methods get stuck in local optima frequently because they rely on local search strategies causing instability prone to divergence from small changes in policies.
All sources
77 references cited across the entry
- 1journalReinforcement Learning: A SurveyLeslie P. Kaelbling et al. — 1996
- 2bookReinforcement Learningvan Otterlo, M. et al. — 2012
- 3bookReinforcement Learning for Sequential Decision and Optimal ControlShengbo Li — 2023
- 4bookArtificial intelligence: a modern approachStuart J. Russell et al. — Prentice Hall — 2010
- 5journalNeural Basis of Reinforcement Learning and Decision MakingDaeyeol Lee et al. — 21 July 2012
- 6journalCommunity energy storage operation via reinforcement learning with eligibility tracesEdgar Mauricio Salazar Duque et al. — 2022
- 7arxivALLSTEPS: Curriculum-driven Learning of Stepping Stone SkillsZhaoming Xie — 2020
- 8journalOptimal dispatch of PV inverters in unbalanced distribution systems using Reinforcement LearningPedro P. Vergara et al. — 2022
- 9journalSelf-Learned Intelligence for Integrated Decision and Control of Automated Vehicles at Signalized IntersectionsYangang Ren et al. — 2026
- 10bookSimulation-based Optimization: Parametric Optimization Techniques and ReinforcementAbhijit Gosavi — Springer — 2003
- 11citationOptimal adaptive policies for Markov Decision ProcessesApostolos N. Burnetas et al. — 1997
- 12citationKI 2011: Advances in Artificial IntelligenceMichel Tokic et al. — Springer — 2011
- 14journalReinforcement learning with replacing eligibility tracesSatinder P. Singh et al. — 1996-03-01
- 15thesisTemporal Credit Assignment in Reinforcement LearningRichard S. Sutton — University of Massachusetts, Amherst, MA — 1984
- 16journalLearning to predict by the method of temporal differencesSteven J. Bradtke et al. — 1996
- 17thesisLearning from Delayed RewardsChristopher J.C.H. Watkins — King's College, Cambridge, UK — 1989
- 18journalDetection of Static and Mobile Targets by an Autonomous Agent with Deep Q-Learning AbilitiesBarouch Matzliach et al. — 2022
- 19conferenceA class of gradient-estimating algorithms for reinforcement learning in neural networksRonald J. Williams — 1987
- 20conferenceReinforcement Learning for Humanoid RoboticsJan Peters et al. — 2003
- 21webSimple Reinforcement Learning with Tensorflow Part 8: Asynchronous Actor-Critic Agents (A3C)Arthur Juliani — 2016-12-17
- 22bookA Survey on Policy Search for RoboticsMarc Peter Deisenroth et al. — NOW Publishers — 2013
- 23conferenceIntegrated Architectures for Learning, Planning and Reacting based on Dynamic ProgrammingRichard Sutton — 1990
- 24conferenceSelf-improving reactive agents based on reinforcement learning, planning and teachingLong-Ji Lin — 1992
- 25citationChapter 7 - Meta-reinforcement learningLan Zou — Academic Press — 2023-01-01
- 26conferenceWhen to use parametric models in reinforcement learning?Hado van Hasselt et al. — 2019
- 28journalEfficient Model Learning Methods for Actor–Critic ControlIvo Grondman et al. — 2012-06-01
- 30conferenceReinforcement Learning with Temporal Logic RewardsXiao Li et al. — 2017
- 31journalReward Machines: Exploiting Reward Function Structure in Reinforcement LearningRodrigo Toro Icarte et al. — 2022
- 32journalA probabilistic argumentation framework for reinforcement learning agentsRégis Riveret et al. — 2019
- 33arxivEntity-Centric Reinforcement Learning for Object Manipulation from PixelsHaramati, Dan et al. — 2024
- 34conferenceEntity-based Reinforcement Learning for Autonomous Cyber DefenceIsaac Symes Thompson et al. — ACM — 2024-11-07
- 35webEntity-Based Reinforcement LearningClemens Winter — 2023-04-14
- 36arxivReinforcement Learning with Feedback from Multiple Humans with Diverse SkillsTaku Yamagata et al. — 2021-11-16
- 37journalHierarchical Deep Reinforcement Learning: Integrating Temporal Abstraction and Intrinsic MotivationTejas D. Kulkarni et al. — Curran Associates Inc. — 2016
- 39book2020 Design, Automation & Test in Europe Conference & Exhibition (DATE)Somdip Dey et al. — March 2020
- 40webSmartphones get smarter with Essex innovationTony Quested
- 41webFuture smartphones 'will prolong their own battery life by monitoring owners' behaviour'Rhiannon Williams — 2020-07-21
- 42bookEmbodied Artificial IntelligenceF. Kaplan et al. — Springer — 2004
- 43journalKeep your options open: an information-based driving principle for sensorimotor systemsA. Klyubin et al. — 2008
- 44bookIntrinsically Motivated Learning in Natural and Artificial SystemsA. G. Barto — Springer — 2013
- 45journalDeep Execution - Value and Policy Based Reinforcement Learning for Trading and Beating Market BenchmarksKevin Dabérius et al. — 2020
- 46journalSelf-organizing maps for storage and transfer of knowledge in reinforcement learningThommen George Karimpanal et al. — 2019
- 47harvnbSutton, Barto (2018) p. Section 5.4, p. 100Sutton, Barto — 2018
- 48journalDistributional Soft Actor-Critic: Off-policy reinforcement learning for addressing value estimation errorsJ Duan et al. — 2021
- 49book2020 IEEE 23rd International Conference on Intelligent Transportation Systems (ITSC)Y Ren et al. — 2020
- 50journalDistributional Soft Actor-Critic with Three RefinementsJ Duan et al. — 2025
- 51bookDynamic, Genetic and Chaotic Programming: The Sixth-Generation Computer Technology SeriesBranko Soucek — John Wiley & Sons, Inc — 6 May 1992
- 52journalAn Introduction to Deep Reinforcement LearningVincent Francois-Lavet — 2018
- 53journalHuman-level control through deep reinforcement learningVolodymyr Mnih — 2015
- 54journalExplaining and Harnessing Adversarial ExamplesIan Goodfellow et al. — 2015
- 55bookMachine Learning and Data Mining in Pattern RecognitionVahid Behzadan et al. — 2017
- 56bookAdversarial Attacks on Neural Network PoliciesSandy Huang et al. — 2017-02-07
- 57journalDeep Reinforcement Learning Policies Learn Shared Adversarial Features Across MDPs.Ezgi Korkmaz — 2022
- 58bookProceedings of 1994 IEEE 3rd International Fuzzy Systems ConferenceH.R. Berenji — IEEE — 1994
- 59book2017 IEEE 15th International Symposium on Applied Machine Intelligence and Informatics (SAMI)David Vincze — IEEE — 2017
- 60bookProceeding ICML '00 Proceedings of the Seventeenth International Conference on Machine LearningA. Y. Ng et al. — Morgan Kaufmann Publishers — 2000
- 61journalMaximum entropy inverse reinforcement learningBrian D. Ziebart et al. — AAAI Press — 2008-07-13
- 62journalTrajectory modeling via random utility inverse reinforcement learningAnselmo R. Pitombeira-Neto et al. — March 2024
- 63journalA practical guide to multi-objective reinforcement learning and planningHayes C, Radulescu R, Bargiacchi E, et al — 2022
- 64bookMultiple Attribute Decision Making: Methods and ApplicationsGwo-Hshiung Tzeng et al. — CRC Press — 2011
- 65journalA review of safe reinforcement learning: Methods, theories and applicationsShangding Gu et al. — 10 September 2024
- 66journalA comprehensive survey on safe reinforcement learningJavier García et al. — 1 January 2015
- 67journalImplicit Quantile Networks for Distributional Reinforcement LearningWill Dabney et al. — PMLR — 2018-07-03
- 68journalRisk-Sensitive and Robust Decision-Making: a CVaR Optimization ApproachYinlam Chow et al. — Curran Associates, Inc. — 2015
- 70journalOptimizing the CVaR via SamplingAviv Tamar et al. — 2015-02-21
- 71journalEfficient Risk-Averse Reinforcement LearningIdo Greenberg et al. — 2022-12-06
- 73webAn API for reinforcement learningJanuary 22, 2025
- 74journalDeepSeek-R1 incentivizes reasoning in LLMS through reinforcement learningDeepSeek-AI — January 22, 2025
- 75journalImplementation Matters in Deep RL: A Case Study on PPO and TRPOLogan Engstrom et al. — 2019-09-25
- 76journalDistributional Soft Actor-Critic with Three RefinementsCédric Colas — 2019-03-06
- 77journalReward Machines: Exploiting Reward Function Structure in Reinforcement LearningIdo Greenberg et al. — PMLR — 2021-07-01