Seeing in Point Clouds

Robotic grasping research runs on datasets that are quietly unrealistic. CAD libraries are too pristine to survive a real sensor; existing scan collections lack texture, need exotic rigs, or use objects nobody else can buy. The benchmarks that 3D networks are tuned on are dominated by texture-less shapes — so models post impressive leaderboard numbers without ever being asked the question that matters for grasping: can you pick out one specific, textured item among visually similar distractors?

The design philosophy was radical reproducibility. Eighty standard grocery items, each identified by its EAN barcode so any lab in the world can rebuild the physical benchmark with a shopping list. Capture runs on the two most common depth cameras in robotics — no exotic scanners. At every 5-degree step of a motorized table, eight cameras fire under three lighting intensities, baking real sensor noise and light variation directly into 1,200 partial views per object.

A processing pipeline then composites targets with randomized distractor objects into labeled scenes at any resolution — including an adversarial test where the distractor shares the target’s shape, size, and dominant color, so only genuine texture understanding can tell them apart.

Only 4 of 11 architectures could even partially segment the target among distractors in the real world. The pattern was predictive: networks proven on shape-centric benchmarks transferred to texture-rich real data, while indoor-scene specialists — including celebrated transformer models — collapsed entirely.

A counterintuitive scaling result closed the study: feeding the networks larger samples made them both more accurate and 1.78× faster at inference.

The MSc dissertation carried the question into deployment: a synthetic companion dataset of 12,000 cluttered scenes at a severe 1:9 class imbalance, three architectures trained in a unified PyTorch pipeline, and the best network compressed for real-time use. L1 pruning, INT8 quantisation, and knowledge distillation cut PointWeb’s per-scene latency from 1.91 to 1.77 seconds while preserving ~99% of its 0.986 validation mIoU.