Depth & Point Clouds

A colour camera tells you what is where in the picture; a depth camera tells you where it is in space. Turn depth images into point clouds, clean them up, find the table under a miscalibrated camera, tell the objects apart, measure a box's pose with ICP, and pick the balls from look-alikes of the same colour.

What you'll learn

• >Back-project a depth image into a point cloud in the world frame
• >Crop, voxel-downsample and estimate surface normals with PCA
• >Fit the table plane with RANSAC and measure heights from it
• >Separate objects with Euclidean clustering
• >Find an object's pose with SVD alignment and ICP
• >Recognise objects by fitting shapes, not by colour

Before you start

Take Seeing with a Camera first: this course starts from its pinhole model and camera pose. You should be at ease with numpy arrays, including boolean masks, broadcasting and np.linalg.

Steps

1. 1From depth to pointsTurn every pixel of a depth image into a 3-D point in the world, see the cloud, and point at the tallest thing on the table.Depth imagesBack-projectionPoint cloudsOpen
2. 2Cleaning up a cloudCrop a noisy cloud to the workspace, thin it out with a voxel grid, estimate surface normals, and set a ball on the only flat top.CroppingVoxel downsamplingSurface normals (PCA)Open
3. 3Finding the tableFit the table's plane with RANSAC, even though objects cover it and the camera's calibration is slightly off, then measure heights from it.RANSACPlane fittingCalibration errorPro
4. 4Telling objects apartSplit the points above the table into separate objects with Euclidean clustering, measure each one, and point at them in turn.Euclidean clusteringFlood fillBounding boxesPro
5. 5Where is the box?Align a model of a box with its points (best-fit rotation by SVD, then ICP) to find its pose, and trace its top with the gripper.Pose estimationKabsch / SVD alignmentICPPro
6. 6Pick by shapeEvery ball has a twin of the same colour and width. Fit a sphere to each object's points, keep the ones that really are balls, and put them all in the cup.Sphere fittingLinear least squaresShape-based pickingPro
7. ★Bonus: Let the table fix the cameraUse the fitted table to correct the camera's pose itself, so every point's z is its true height, and see which errors a plane can't reveal.Self-calibrationRotation between two vectorsObservabilityPro