Robot Learning

Every controller so far was written by hand. Here the robot learns one: plan around a wall with value iteration, learn the same plan by trial and error, copy an expert and fix the copy's drift, train a throw from its own attempts, and learn a model of a cart-pole to plan with.

What you'll learn

• >Write a robot task as a Markov decision process, and evaluate a policy on it
• >Plan the best policy with value iteration
• >Learn action values from experience with Q-learning and ε-greedy exploration
• >Copy an expert by behaviour cloning, and fix its drift with DAgger
• >Train a random policy with REINFORCE, and tame its noise with a baseline
• >Learn a dynamics model from data and plan on it with the cross-entropy method
• >Plan on a model learned from counts, and compare it with Q-learning on the same moves (bonus)

Before you start

You should be at ease with numpy arrays, broadcasting and loops. Least squares (Linear Algebra, Probability & Optimisation) helps, and Pick & Place and Balancing & Optimal Control introduce the arm and the cart-pole, but each step explains the learning maths it uses.

Steps

1. 1The reach as a decision processCut the arm's configuration space into a grid of states, write a policy that lifts the hand over the wall, and measure how good it is with policy evaluation.Markov decision processPolicyReturn and valuePolicy evaluationOpen
2. 2Planning with value iterationBack up the Bellman optimality equation until the values settle, watch them spread out from the goal, and drive the arm over the wall from four starts.Bellman optimality equationAction valuesContractionGreedy policyOpen
3. 3Learning from trial and errorForget P and R: learn action values from the arm's own moves with Q-learning and ε-greedy exploration, and watch the success rate climb.Q-learningTemporal-difference targetε-greedy explorationLearning curvePro
4. 4Copying an expertFit a reaching policy to an expert's demonstrations by least squares, and see why the features that fit the data best park the arm short of the target.Behaviour cloningLeast squaresFeaturesTraining error versus closed loopPro
5. 5When the copy driftsWatch a cloned lane-following policy stray into states the expert never showed it, measure how far, and fix it with DAgger.Distribution shiftCompounding errorsNearest-neighbour distanceDAggerPro
6. 6Learning to throwTrain a throw into the cup from its own attempts with the REINFORCE policy gradient, and see how a baseline tames the gradient's noise.Stochastic policyPolicy gradientBaselineVariancePro
7. 7Learning a model to plan withLearn the cart-pole's dynamics from random pushes, compare two models, and catch the falling pole by planning on the better one with the cross-entropy method.Model-based RLLearned dynamicsCross-entropy methodModel predictive controlPro
8. ★Bonus: Learn the model, then planBack on the grid: count what the moves did to learn P and R, plan on that model with value iteration, and beat Q-learning on the same experience.Model-based RLMaximum-likelihood modelSample efficiencyPro