Copying an expert

An expert to copy

Q-learning needed hundreds of thousands of moves. Often there's an expert to watch instead: a person with a joystick or, here, the damped least-squares controller from Jacobians & Velocity Control. rl.expert_demos(20, seed=0) records 20 of its reaches. Each row of obs (N × 7) is an observation $o = [q_0, q_1, q_2, q_3, e_x, e_y, e_z]$, the joint angles (rad) and then $e$ = target − gripper (m); the same row of actions (N × 4) is the expert's joint velocity (rad/s). Behaviour cloning fits a policy to those pairs: plain supervised learning.

Least squares

Take a linear policy $\pi(o) = \phi(o)^\top W$: features $\phi(o)$, $k$ numbers worked out from $o$, times weights $W$ ($k \times 4$). Stack every row's features into $X$ ($N \times k$) and the actions into $Y$ ($N \times 4$). Least squares picks the $W$ that minimises $\|XW - Y\|^2$, the sum of the squared action errors. Setting its gradient to zero gives the normal equations

a $k \times k$ system: np.linalg.solve(X.T @ X, X.T @ Y).

A better fit isn't a better controller

The program fits two feature sets. The given features_everything is the whole observation and a constant 1. Yours, features_error, is $e$ alone. Everything contains the error, so it fits the demonstrations at least as well: its training RMS (root mean square action error) is lower. But on the target, where $e = 0$, its $q$ terms and the 1 don't vanish, so the policy keeps commanding a little motion, and the arm settles where the error terms cancel it: short of the target. With error features the policy is exactly zero there, as the expert is. Only running the policy, closed loop, shows that.

Your task

1. Write least_squares(X, Y) by the normal equations.
2. Write features_error(obs).
3. Write fit(obs, actions, features): build $X$, solve for $W$, and return policy(o), which gives features(o) @ W.

The arm then reaches for 6 new targets with everything and 10 with your features, 2 s each; the Plots tab shows the distance. 8 of your 10 must end within 1 cm.