LLM-Supervised Point Cloud Processing: From Unsupervised 3D Scene-Graph Generation to Interactive Scene Manipulation
TL;DR
An end-to-end pipeline for 3D scene understanding is demonstrated which integrates unsupervised graph-based point cloud segmentation with LLM-enabled spatial reasoning and editing and achieves a 100% constraint-satisfaction rate and outperforming a human annotator.
Abstract
Abstract. We demonstrate an end-to-end pipeline for 3D scene understanding which integrates unsupervised graph-based point cloud segmentation with LLM-enabled spatial reasoning and editing. A point cloud is segmented into a SemanticPatch decomposition (stage 1), labeled using a zero-shot vision-language model (stage 2; SAMv2, CLIP), encoded into a scene graph in the latent space (stage 3) capturing geometry, topology, and constraints, and finally manipulated by an LLM-based agent (stage 4) to execute a specified editing task. The LLM agent can be instructed by natural language input to reason about a scene graph and a point cloud, compute a geometric transformation for the input point cloud, and check its own output against a set of constraints (e.g. ADA-compliance). We validate our approach on three different point clouds: a classroom (Leica RTC360, 1.3 M points), a construction site (NavVis VLX mobile scanner, 4.4M points), and the Paris-Lille-3D benchmark. Our segmentation approach scores 97–99% on the fitness score and 92–99% on the F1-score across all three benchmarks. Our LLM agent solves reconfiguration tasks in 1–10 min, achieving a 100% constraint-satisfaction rate and outperforming a human annotator.