Policy gradient (REINFORCE)

What it computes

REINFORCE improves a random policy from the rewards of its own trials alone: no model, and no derivative of the reward. Here the policy is a Gaussian over the throw's parameters $\theta = [\text{aim}, \text{power}, \text{release}]$. Each trial draws $\theta \sim \mathcal N(\mu, \sigma^2 I)$ with a fixed σ, and learning moves the mean μ.

The likelihood-ratio trick

$R(\theta)$ is a simulation, so to climb $J(\mu) = \mathbb E[R(\theta)]$ move the derivative onto the policy, using $\nabla \pi = \pi \nabla \log \pi$:

A batch of $N$ samples estimates it:

The baseline $b$ cuts the variance without changing the mean, because $\mathbb E[\nabla \log \pi] = 0$ (the batch mean only scales $g$ by $\tfrac{N-1}{N}$). Every reward here is negative, so without $b$ every sample pushes μ away from itself. With $b$, better-than-average throws pull and worse ones push.

Why it's finicky

$g$ is noisy: with a fixed step, μ jitters around the optimum or jumps off the smooth part of the landscape. So the step decays, $\alpha_k = 4 \cdot 0.96^k$. Karpathy left REINFORCE out of REINFORCEjs as too finicky. It works here because a whole throw is one action with one reward, and θ has three parameters, each scaled to move the landing a few centimetres.

mu = theta0
repeat iters times:
    thetas = mu + sigma * randn(batch, 3)
    R = [rollout(theta) for theta in thetas]
    mu = mu + lr * policy_gradient(thetas, R, mu, sigma)
    lr = lr * decay

Tools

rl.rollout(theta) simulates a throw off-screen (3.5 ms) and returns minus the distance (m) from the cup's centre to where the ball comes down. Each call counts toward the budget. The program trains from θ = 0, plots the misses and throws once with your μ.

Next: actor-critic methods learn the baseline as a value function, and PPO limits how far each update moves the policy.

Your task

Implement policy_gradient(thetas, rewards, mu, sigma) (thetas is N×d) and reinforce. The grader checks $g$ exactly on fixed batches, replays your rollouts to check each update, and reruns train(seed) for seeds 3, 17, 101: every throw must land in the cup within 300 rollouts.