Deeply Guided Lightweight 3D Scene Understanding Method and Its Application in Mobile Robot Perception
TL;DR
A depth-guided lightweight multi-task 3D scene understanding method based on a lightweight U-Net architecture, uses RGB and depth four-channel dual- modal input, and constructs a network structure with a shared encoder and independent decoders, allowing a single inference to simultaneously accomplish the three core tasks of semantic segmentation, depth completion, and obstacle detection.
Abstract
To address the core issues faced by edge-side 3D scene perception of mobile robots, such as the difficulty of balancing accuracy and real-time performance, insufficient multi-task fusion, and deviations in depth geometric consistency, this paper proposes a depth-guided lightweight multi-task 3D scene understanding method. This method is based on a lightweight U-Net architecture, uses RGB and depth four-channel dual- modal input, and constructs a network structure with a shared encoder and independent decoders, allowing a single inference to simultaneously accomplish the three core tasks of semantic segmentation, depth completion, and obstacle detection. It also Introduce depth consistency error (DCE) to construct a weighted joint loss function, strengthening scene geometry structure learning; through a dual lightweight strategy of structured pruning and INT8 quantization, precisely adapt to NVIDIA Jetson Xavier NX edge hardware. Experimental results on the NYU Depth V2 and KITTI datasets show that this method achieves a semantic segmentation mIoU ≥ 60%, depth completion RMSE ≤ 1.0m, obstacle detection F1-Score ≥ 85%, and edge- end inference speed ≥ 15 FPS, effectively balancing perception accuracy and real-time performance, providing a highly practical lightweight solution for autonomous perception in mobile robots.