When the copy drifts

Mistakes compound

In step 4 the cloned arm went where the expert had gone, so the data covered it. Now the rover follows a lane by lidar: a policy turns 9 ranges, from its right to its left, into a turn rate, and the expert is a pure-pursuit controller that knows the rover's true pose. rl.lane_demos(1) records one clean lap by the expert, and fit (step 4's least squares, on 55 quadratic features of the ranges) copies it closely.

Each drive starts somewhere new, a little off the centre line and askew. A small error takes the rover somewhere the expert never was. There the fit is guesswork, so the next error is bigger, and so on into a wall. That is distribution shift: the policy is tested on the states it visits, not on the expert's.

Measuring how new a state is

The novelty of an observation $o$ is its distance to the nearest observation in the training data $D$:

The program plots each drive's largest novelty. The drives that end at a wall are the ones that strayed furthest.

DAgger

DAgger (dataset aggregation) collects data where the policy actually goes. In round $i$, drive the current policy $\pi_i$ and record every observation it reaches, with the action the expert would have taken there, $a^*$. drive(policy, beta) returns both. Add these pairs to the data and refit: the new policy has seen its own mistakes, labelled. With probability $\beta_i = 0.5^i$ the expert steers instead, so the early drives get somewhere.

Your task

1. Write novelty(obs, seen), one distance per row of obs. Broadcasting obs[:, None, :] - seen[None, :, :] gives every pair at once.
2. Write dagger_round(policy, obs, actions, drive, fit, beta, episodes). np.vstack and np.concatenate add rows.

The program clones, runs 3 rounds of 3 drives, then tests on 5 new starts. At least 4 must travel 0.8 m without touching a wall.