From pixels back to the table

Pixels are rays

Projection throws depth away: every point on a line through the lens lands on the same pixel. So going backwards, a pixel doesn't give you a point. It gives you a ray:

$K^{-1}[u, v, 1]^\top$ is the ray's direction in the camera frame, scaled so that $z_c = 1$. $R$ turns it into world coordinates, and the ray starts at the camera's position $t$.

A known plane fixes the depth

One camera can't tell how far along the ray a ball is. But you know something extra: every ball rests on the table, so its centre is at height $z = r = 0.025$ m. Walk along the ray until you reach that height:

The camera looks down, so $d_z$ is negative, and so is $r - o_z$. That makes $s$ positive. Warehouse robots use this trick to find boxes on a conveyor, and cars use it to place lane markings on the road.

Another route: depth

img.depth[v, u] is the distance along the camera's $z$ axis to whatever that pixel sees. Scaling $K^{-1}[u, v, 1]^\top$ by it gives the 3-D point directly in the camera frame, even off the table, and $R$ and $t$ then carry it into the world. But it lands on the ball's surface, not its centre. Depth cameras and stereo rigs give you that image; the plane trick needs only an ordinary camera.

Your task

Implement pixel_to_world(u, v) so it returns np.array([x, y, z]) on the plane $z = 0.025$. The starter finds each ball with a step-2 find_ball and prints your estimate next to the truth.

The grader checks 20 pixels, and each answer must be within 5 mm.