Seeing with a Camera

Give the robot a camera and take ground truth away. Model the camera, find balls in images, turn pixels back into table positions, pick by sight, and calibrate the camera from robot motions.

What you'll learn

• >Model a camera with intrinsics and extrinsics
• >Project world points to pixels
• >Find coloured objects in an image with numpy masks
• >Back-project pixels to 3-D using a known plane
• >Pick objects using vision only
• >Calibrate a camera's pose from robot motions

Before you start

You should be comfortable with basic Python: variables, loops and functions. We'll introduce the robotics and the maths as you go. The first visit downloads the physics engine and Python, about 20 MB, and later visits load from your browser's cache.

Steps

1. 1The pinhole cameraPredict where a 3-D point shows up in the camera image, using the camera's pose and its intrinsics.Pinhole modelIntrinsicsExtrinsicsOpen
2. 2Finding balls by colourTurn an image into masks with numpy, then locate each ball as the centroid of its pixels.Colour thresholdsMasksCentroidsPro
3. 3From pixels back to the tableTurn a pixel into a ray and intersect it with the plane the balls rest on to get a 3-D position.Back-projectionRaysPlane intersectionPro
4. 4Pick what you seeClose the loop: find each ball with the camera, turn it into a position and pick it, with no ground truth at all.Perception pipelineOcclusionClosed-loop pickingPro
5. ★Bonus: Where is the camera?The camera got bumped. Recover its pose from robot motions: marker pixels plus known 3-D points, solved with a DLT.Hand–eye calibrationDirect linear transformSVDPro