The pinhole camera

From positions to pixels

In Pick & Place, world.balls() handed you exact positions. Real robots have to see. A camera now stands at the far end of the table, facing the robot. camera.capture() returns an Image whose rgb is a 240 × 320 × 3 array of bytes. Pixel $(u, v)$ is column $u$ and row $v$, so its colour is img.rgb[v, u].

Before a robot can find things in a picture, it has to know where things would appear. That's the pinhole model, and it has two steps.

1. Into the camera's frame

camera.pose is a 4 × 4 transform from camera to world coordinates. Its rotation $R$ = T[:3, :3] holds the camera's axes as columns, and $t$ = T[:3, 3] is the camera's position. Inverting it moves a world point $p$ into the camera frame:

$$p_c = R^\top (p - t)$$

Camera axis	Points
$x$	right in the image
$y$	down in the image
$z$	forward, out of the lens

2. Onto the image

The intrinsic matrix $K$ (camera.K) scales by the focal length $f$, measured in pixels, and shifts by the principal point $(c_x, c_y)$:

$$\lambda \begin{bmatrix} u \\ v \\ 1 \end{bmatrix} = K\,p_c, \qquad K = \begin{bmatrix} f_x & 0 & c_x \\ 0 & f_y & c_y \\ 0 & 0 & 1 \end{bmatrix}$$

Dividing by the depth $\lambda = z_c$ gives $u = f_x\,x_c / z_c + c_x$ and $v = f_y\,y_c / z_c + c_y$. That division is why distant things look small.

Your task

Implement project(p) so it returns np.array([u, v]). The starter captures an image and draws a white cross wherever project says each ball is. Open the Images tab: with a correct project, every cross sits on its ball.

The grader compares project with the real camera on 20 random points. Each must be within 0.5 px.