Dense-to-Sparse Knowledge Distillation for 3-D Semantic Segmentation

Abstract

Single-frame LiDAR and RGB-D point clouds are inherently sparse: occlusions, limited field of view, and sensor resolution leave many objects geometrically incomplete in any individual capture. While multi-frame aggregation can reconstruct denser scene geometry, it increases inference cost and introduces motion artifacts in dynamic scenes. In this work, we propose a training framework that obtains the benefits of dense geometric supervision without any inference time overhead. During training, a teacher network processes dense point clouds constructed by aggregating points by semantic class within each Waymo scene or cropping scene reconstructions on ScanNet. A student network, operating on standard sparse single-frame inputs, is supervised through a point-count-independent distillation objective that aligns per-channel feature statistics. This formulation is well-defined regardless of differences in the number of points between teacher and student feature maps, and leaves the student architecture and inference pipeline unchanged. We evaluate our approach on two benchmarks spanning complementary modalities: the Waymo Open Dataset (outdoor LiDAR) and ScanNet (indoor RGB-D), using both a sparse convolutional backbone (SpUNet) and a transformer-based architecture (Point Transformer v3). Across both settings, distillation consistently improves over the sparse single-frame baseline, with gains of up to + 3.62 mIoU on Waymo and + 2.84 mIoU on ScanNet. Note that ScanNet results are reported in an online single-frame regime–one unprojected RGB-D frame at a time, without access to a pre-reconstructed scene–which is harder than the standard offline scene-level evaluation and explains the lower absolute mIoU values.