The reach as a decision process

The task as a decision process

The arm must carry its hand from the ball side of the table to above the cup, and the Motion Planning wall is in the way. Here it thinks in a grid, a 24 × 24 slice of its configuration space. Rows are the shoulder angle (row 0 holds the hand high, row 23 brushes the table), columns the base angle (column 0 far left, column 20 facing the cup); the elbow and wrist stay fixed. Each cell is a state $s = 24i + j$, for row $i$ and column $j$.

A Markov decision process (MDP) describes the task:

Symbol	Here
$a$	an action: one of 8 moves to a neighbouring cell, UP, DOWN, LEFT, RIGHT or a diagonal
$P(s' \mid s, a)$	the chance that $a$ takes the arm from $s$ to $s'$: the intended cell 90 % of the time, otherwise a slip 90° to one side
$R(s, a)$	the expected reward: −1 per move, −10 for a move into something (the arm stays put)
$\gamma$	the discount, 0.95: a reward $k$ moves away counts $\gamma^k$ times

The goal, cell (17, 20) above the cup, ends the task: nothing more is earned there.

Policies and their value

A policy $\pi$ picks an action $\pi(s)$ in every state. Its value $V^\pi(s)$ is the expected return $r_0 + \gamma r_1 + \gamma^2 r_2 + \dots$ (the discounted sum of the rewards $r_t$) when you start in $s$ and follow $\pi$: minus a discounted count of moves, so closer to 0 is better. It obeys the Bellman equation

$$V^\pi(s) = R(s, \pi(s)) + \gamma \sum_{s'} P(s' \mid s, \pi(s))\, V^\pi(s')$$

Policy evaluation turns it into an update: start from $V = 0$ and apply the right-hand side to every state, sweep after sweep, until no value changes by more than a tolerance. In numpy, P[np.arange(S), policy] picks each state's row for its own action, an S × S matrix $P_\pi$, and R[np.arange(S), policy] the rewards $R_\pi$. One sweep is R_pi + gamma * P_pi @ V.

Your task

1. Write evaluate_policy(P, R, gamma, policy, tol). mdp.P is (S, A, S) and mdp.R is (S, A).
2. Fix my_policy(i, j). It heads straight for the cup's column, and the wall stops it. Make it lift the hand over the wall first: rows 0 to 10 are clear all the way across.

The Images tab shows your $V^\pi$ (blue low, yellow high, blocked cells black) with the path from start 1, and the arm drives it. From all three starts your policy must reach the goal without touching anything.