Learning to throw

Reward alone

Cloning needed an expert and Q-learning a small table of states. The throw has neither. Three numbers $\theta$ = [aim, power, release] decide a throw, and the only feedback is the reward: minus the distance (m) from the cup's centre to where the ball comes down. rl.rollout(theta) simulates a throw off-screen and returns its reward; rl.throw(theta) throws the real ball, once.

A random policy, and its gradient

REINFORCE makes the policy random: $\theta \sim \mathcal{N}(\mu, \sigma^2 I)$, a Gaussian around a mean $\mu$ with the same spread $\sigma$ in every parameter. It climbs the expected reward $J(\mu) = \mathbb{E}[R(\theta)]$ by gradient ascent. $R$ can't be differentiated, but the policy's log-probability can: $\nabla_\mu \log p(\theta) = (\theta - \mu)/\sigma^2$, the score. That gives

for $N$ samples $\theta_i$ with rewards $R_i$. Samples that did better than $b$ pull $\mu$ towards them; worse ones push it away. Then $\mu \leftarrow \mu + \alpha g$, with learning rate $\alpha$.

The baseline

Any constant $b$ leaves the expectation alone, because $\mathbb{E}[\theta - \mu] = 0$, but it changes the noise. With $b = 0$ every throw misses, so every $R_i$ is negative and every sample pushes $\mu$ away from itself, good or bad; only the small differences between rewards carry information, buried under that push. The baseline $b$ = the batch's mean reward keeps just the differences. The Plots tab shows 20 estimates at $\theta = 0$, made from the same throws both ways.

Your task

1. Write policy_gradient(thetas, rewards, mu, sigma, baseline), with $b$ = rewards.mean() if baseline else 0.
2. Write reinforce(rollout, mu, baseline). ITERS times: draw BATCH samples mu + SIGMA * np.random.randn(BATCH, 3), roll each out with rollout, step mu += lr * g and then lr *= DECAY, with lr starting at LR. Return mu.

The program trains twice from $\theta = 0$ with the same random numbers, without and with the baseline, and throws the real ball with the baseline's $\mu$. It must land in the cup.