Linear systems: calibrate a camera

A camera that speaks pixels

A new camera, overhead, hangs high above the table looking straight down. It finds every ball, but it reports pixels $(u, v)$: column $u$ and row $v$ of a 640 × 480 picture. Where the camera hangs and how it's turned change every run, and world.balls() is switched off. To use the camera you have to calibrate it: work out how pixels turn into metres.

The map is linear

Because the camera looks straight down, points at one height (such as the balls' centres, 2.5 cm up) appear as a scaled, rotated and shifted copy of the table:

$$x = a\,u + b\,v + c, \qquad y = d\,u + e\,v + f$$

That's six unknown numbers, $a$ to $f$. Each is only multiplied by something you know ($u$, $v$ or 1) and added up, so these are linear equations. In matrix form, with $A$ a 2×3 matrix:

$$\begin{bmatrix} x \\ y \end{bmatrix} = A \begin{bmatrix} u \\ v \\ 1 \end{bmatrix}, \qquad A = \begin{bmatrix} a & b & c \\ d & e & f \end{bmatrix}$$

Three points pin it down

Put the gripper on a pad whose table position $(x, y)$ you know, and overhead.marker() gives the pixel of a marker at the grasp point. The map only holds at one height (things closer to the camera look bigger), so read the marker with the grasp point at ball-centre height, where the pads are. Each pad gives two equations; three pads give six, one for each unknown. The x-equations only involve $a$, $b$ and $c$:

$$\underbrace{\begin{bmatrix} u_1 & v_1 & 1 \\ u_2 & v_2 & 1 \\ u_3 & v_3 & 1 \end{bmatrix}}_{M} \begin{bmatrix} a \\ b \\ c \end{bmatrix} = \begin{bmatrix} x_1 \\ x_2 \\ x_3 \end{bmatrix}$$

The y-equations use the same $M$ with $d$, $e$, $f$ and the $y_i$. A square system $M\mathbf{w} = \mathbf{r}$ (unknowns $\mathbf{w}$, known right-hand side $\mathbf{r}$) has exactly one answer when its equations are independent, and np.linalg.solve(M, r) finds it.

If the three pads lie on one straight line, one row of $M$ is a mix of the other two. The equations repeat each other, $M$ is singular, and there's no single answer. Keep calibration points spread out.

Seeing with a Camera, step 1 builds the full camera model behind this map.

Your task

1. Write solve_affine(pixels, points). pixels holds three [u, v] rows and points three [x, y] rows. Return the 2×3 matrix $A$.
2. The starter visits the three yellow pads (world.targets()) at ball height and records the marker's pixel on each.
3. Convert the red ball's pixel (overhead.balls()["red"]) to table coordinates with $A$, and put the ball in the cup.