Telling objects apart

One cloud, many objects

With the table gone, what's left is a few hundred points in one array. You can see that they form separate objects. The robot can't, until something groups them.

The rule is simple: two points belong to the same object if you can walk from one to the other in steps shorter than a radius $r$, stepping only on points. That's Euclidean clustering. Pick $r$ larger than the gaps within an object (1 cm voxels, so neighbours sit about 1 cm apart) and smaller than the gaps between objects. Here $r = 1.5$ cm.

Flood fill

Clusters grow like paint spilled on a floor:

1. Take a point no cluster has reached yet: it starts a new cluster, and goes on a frontier list.
2. Pop a point off the frontier. Every unreached point within $r$ of it joins the cluster and the frontier.
3. Repeat until the frontier is empty: that cluster is complete. Go back to 1.

Keep a seen array so each point is reached once. Clusters with fewer than min_size points are leftover noise, not objects: label their points $-1$. Number the real ones $0, 1, 2, \dots$ as you find them.

Each step compares one point with all the others, $N$ distances at a time with numpy. For a few hundred points that's instant. For millions you'd use a grid or a k-d tree to find neighbours.

Measuring each object

The camera sees only an object's near side, so the mean of its points sits closer to the camera than its middle. The middle of its bounding box, halfway between the smallest and largest $x$, $y$ and $z$, does better: the camera sees the whole top of a box or a can, so the bounding box is centred over it. A ball's far half stays hidden, and its bounding box still falls about a centimetre short (step 6 fixes that). The box's sides give the object's size.

Your task

Write cluster(points, radius, min_size) returning one label per point. The program draws each object in its own colour inside its box, stores objects as (centre, size) pairs, and points the gripper at each one in turn.