Behaviour cloning

What it computes

Behaviour cloning learns a policy by copying an expert: record what it saw, $o_i$, and what it did, $a_i$, then fit $\pi(o_i) \approx a_i$. It is plain supervised learning. The expert's actions are the labels, so no reward and no trial and error are needed.

Least squares on features

Make the policy linear in features: $\pi(o) = \phi(o)^T W$. Stack the rows $\phi(o_i)^T$ into $\Phi$ and the actions into $A$, and pick the best fit:

X = [features(o) for o in obs]                 # N x k
W = least-squares solution of X W = actions    # k x 4
policy(o) = features(o) @ W

Choosing the features

An observation is $o = [q_0, q_1, q_2, q_3, e_x, e_y, e_z]$: the joint angles, then $e$ = target − gripper (m). Use $\phi(o) = e$: then the policy is exactly zero on the target, where the expert stops too. Regress on all seven numbers and the $q$ terms don't vanish at $e = 0$, so the arm parks about 1 cm short (1–7 of 20 targets instead of 20). Try it and watch the plot.

Why it can fail

The policy only learned the states the expert visited. Its small mistakes lead it to states nobody demonstrated, where it errs more: errors compound (distribution shift). DAgger fixes this: run the policy, have the expert label the states it reaches, add them to the data and refit.

Tools

rl.expert_demos(40, seed=0) returns (obs, actions), 2,400 rows of 7 and 4 numbers. np.linalg.lstsq(X, Y, rcond=None)[0] solves least squares.

See it in context

The expert is the damped least-squares step from the Kinematics set's Numerical inverse kinematics, run as a controller: $\dot q = K J^T (J J^T + \lambda^2 I)^{-1} e$.

Your task

Implement features(obs) for one observation and fit(obs, actions), which returns the policy: a function from one observation to 4 joint velocities (rad/s). It must bring the gripper within 1 cm of at least 16 of 20 targets it has never seen. The grader also tests fit on made-up data.