Pick what you see

Perception closes the loop

world.balls() is switched off. From now on the robot knows only what the camera shows it. Every picking robot runs the same loop:

1. Sense: take a picture.
2. Estimate: turn pixels into positions with find_ball and pixel_to_world.
3. Act: pick and place.
4. Repeat, starting from a fresh picture.

Get out of your own way

The camera sees the arm too. With the gripper over the table, the arm can hide a ball completely, or cover half of it and drag your centroid off. So always move to a clear-view pose before you look. This step starts in one: arm.home leans the arm back behind its base, out of the camera's view of the table.

Look again before every pick

Fingers brush neighbouring balls, and a missed grasp can push a ball a few centimetres, so an old picture goes stale. Re-capture before each attempt. It costs a second or two of arm motion and saves a lot of misses.

Balls you've already dropped can still show above the cup's rim, and back-projecting them is meaningless because they're not on the table plane. So deal with each colour once: look, pick, and look again only after a miss.

Your task

Put all four balls in the cup using only the camera. The balls move to a new layout on every run.

• pick_and_place(p) is the routine from Pick & Place. It returns False if the grasp missed.
• Write locate(color): clear the view, capture, find the ball and back-project it.
• Loop over the colours, and retry a colour (up to three times) after a miss.

For a bonus, take a fresh picture before every grasp. Press Run a few times: each run is a new layout.