Apprenticeship learning

• 1. Introduction

  ›

  Apprenticeship learning asks a deceptively simple question: what if a machine could learn not from rules, but from watching? In artificial intelligence, the field known as apprenticeship learning, or learning from demonstration, treats observation of an expert as the raw material for building a capable system. The training dataset is not a spreadsheet of labeled examples. It is a record of a skilled agent doing the actual work.

  The idea surfaces across robotics, video games, helicopter aerobatics, and questions as large as how a robot might come to understand human ethics. What unites every approach is the apprentice relationship: a novice system watches, then tries to generalize. How that generalization happens, though, is where the real complexity lies. Some systems map what they see directly to what they should do. Others try to reverse-engineer the goals that must have motivated the expert's behavior. And the gap between those two strategies turns out to matter enormously.

• 2. Mapping What the Expert Does

  ›

  The simplest strategy in apprenticeship learning is to copy. Mapping methods try to form a direct link between the states a system encounters and the actions it should take, or between states and the reward values attached to them. In 2002, researchers applied this approach to teach an AIBO robot basic soccer skills, using observed play to build that direct state-to-action correspondence.

  In robotics demonstrations, the visible result can look almost magical. A human operator takes a robot arm, physically moves it through a task, and the robot reproduces the motion. One illustration often used: a person shows the arm how to place a cup under a coffeemaker and press the start button, and the arm later imitates that behavior step for step. But that surface simplicity is misleading. The internal process is far more complex than what any observer watching the replay can see.

  One of the first formal works on this kind of imitation, where anthropomorphic robots learn by watching, appeared in Adrian Stoica's PhD thesis in 1995. The field was already probing limits that the copy-what-you-see approach could not easily overcome.

• 3. The Pendulum Problem

  ›

  In 1997, robotics researcher Stefan Schaal was working with the Sarcos robot arm on a task that illustrates why mimicry alone runs into trouble. The goal was to solve the pendulum swingup: get the arm to move in a way that drives a pendulum into and through its upright position. The problem can be described with precise mathematical formulas, but solving those formulas directly is genuinely hard.

  Schaal's insight was to skip the brute-force solver entirely. Instead, his team recorded the movements of a human demonstrator. Over three seconds, the angle of the pendulum was logged at the y-axis, against time on the x-axis, producing a trajectory that looked like a pattern on a diagram. The values moved from roughly -3.0 radians at zero seconds, through -2.8 at 0.5 seconds, down to -4.5 at one second, and back up to -1.0 at 1.5 seconds.

  In computer animation, the same principle is called spline animation. The reproduction step turned out to be surprisingly tractable. Given a trajectory over time, the system needs only to find control actions that track the system to that trajectory. This kind of tracking is known as PID control, or steering behavior, because the aim is simply to bring the robot to a given line at each moment in time.

• 4. Reversing the Reward

  ›

  Inverse reinforcement learning, or IRL, takes a fundamentally different angle on the same problem. Standard reinforcement learning gives an agent a reward signal and lets it discover behavior. IRL reverses that direction: it starts with observed behavior and tries to infer what reward function the expert must have been optimizing.

  The formal problem is defined in terms of three inputs. First, measurements of an agent's behavior over time across a variety of circumstances. Second, measurements of the sensory inputs available to that agent. Third, a model of the physical environment, including the agent's body. From those three, IRL attempts to determine the reward function the agent was trying to maximize.

  Driving makes the value of this approach concrete. A driver simultaneously pursues dozens of objectives: maintaining a safe following distance, holding a reasonable speed, avoiding unnecessary lane changes, and more. Writing a single reward function that captures all of those goals explicitly is extremely difficult. But a good driver demonstrates the goals implicitly with every mile. IRL can mine that demonstration for the underlying objectives, even when no one has articulated them.

• 5. AIRP and Helicopter Aerobatics

  ›

  Apprenticeship via inverse reinforcement learning, known as AIRP, was developed in 2004 by Pieter Abbeel, then a researcher associated with Berkeley's EECS department, and Andrew Ng, then an Associate Professor in Stanford University's Computer Science Department. The framework addresses Markov decision processes where no explicit reward function is provided, but where an expert can be observed performing the target task.

  AIRP found particularly striking results in helicopter control. Simple flight trajectories could be worked out intuitively, but aerobatic maneuvers for shows were a different challenge entirely. The approach succeeded on a demanding set of tricks: in-place flips, in-place rolls, loops, hurricanes, and auto-rotation landings. That work was documented by Abbeel, Adam Coates, and Ng in a paper titled "Autonomous Helicopter Aerobatics through Apprenticeship Learning".

  The ability to model reward functions for highly dynamic scenarios, where no obvious reward function exists, is what distinguishes AIRP from simpler mapping approaches. A helicopter aerobatics task involves physics, timing, and competing objectives in a way that resists manual specification.

• 6. Ethics, Cooperation, and Scale

  ›

  IRL researcher Stuart J. Russell has proposed that IRL might eventually be used to observe humans at scale and attempt to codify their complex ethical values. The goal, as Russell frames it, would be creating robots that know, for instance, not to cook a cat, without needing to be explicitly told that rule. The scenario can be modeled as a cooperative inverse reinforcement learning game, where a person player and a robot player work together to secure the person's implicit goals, even though neither party has made those goals explicit.

  In 2017, OpenAI and DeepMind tested deep learning applied to cooperative IRL in relatively contained domains. They worked with Atari games and straightforward robot tasks including backflips. Rather than requiring a full demonstration, the human role was limited to answering queries: given two different actions the robot might take, which is preferred? The researchers found evidence that techniques of this kind may be economically scalable to modern systems, a finding with implications for how far the apprentice model might eventually reach.

• 7. Plans and Preconditions

  ›

  A third major approach to apprenticeship learning works at the level of plans rather than continuous trajectories or reward functions. System model methods attempt to capture world dynamics, while plan-based methods go further: the learner tries to extract rules associating preconditions and postconditions with each action.

  In one 1994 demonstration, a humanoid system learned a generalized plan from watching only two demonstrations of a repetitive ball collection task. Two examples were enough to abstract a reusable strategy. That result points toward one of the central appeals of the apprenticeship framework: a competent demonstration, even a very brief one, can carry more information than a large number of manually specified rules.