LLM-Enhanced Semantic Segmentation of Large-Scale Urban LiDAR Point Clouds via Contextual Prompting

Abstract

Abstract. As a key carrier of 3D spatial information, the semantic segmentation of urban LiDAR point clouds directly impacts the reliability of applications such as autonomous driving and digital twins. However, existing methods face two core bottlenecks: firstly, insufficient adaptation to scene-specific semantics, and secondly, an inference gap between LiDAR structured semantics and segmentation instructions, which makes it difficult to effectively combine the reasoning ability of large language models with LiDAR geometric semantics. To address these issues, this paper proposes a contextual cue framework of "LiDAR semantics-large language model-retrieval-enhanced generation". The framework first designs a lightweight semantic mapping module to convert the structured information inherent to LiDAR into natural language cues that can be understood by LLMs. Secondly, it constructs a LiDAR semantics-text vector library, utilizing the RAG mechanism to retrieve fine-grained knowledge of similar scenes in real-time, generating precise segmentation cues that include geometric features and contextual relationships. Finally, through a three-stage progressive training strategy, it guides LLMs to gradually learn the mapping relationship from semantic understanding to segmentation instruction generation.Ablation experiments verify the effectiveness of each module, and the inference efficiency meets the real-time processing requirements of large-scale urban data. This study provides a new technical path for high-precision, fine-grained semantic segmentation of urban LiDAR point clouds and also offers a theoretical reference for promoting the in-depth application of large language models in 3D spatial intelligence.