SurgTPGS++: Text-promptable Gaussian Splatting with dense features aggregation for semantic 4D surgical scene understanding.

Abstract

With the growing demand for surgical embodied intelligence, accurate semantic understanding of 3D surgical scenes with language-based interaction has become increasingly important. Such capability can assist surgeons in identifying and interacting with surgical instruments and anatomical structures during pre-operative planning and real-time intra-operative guidance. However, existing methods typically address surgical vision-language modeling, 3D reconstruction, and semantic segmentation as separate tasks, leaving real-time language-assisted 3D querying in dynamic surgical scenes largely unexplored. In this paper, we present SurgTPGS++, a novel Gaussian Splatting pipeline for text-promptable 3D surgical scene understanding. We introduce a semantic feature aggregation (SFA) strategy that extracts and integrates rich vision-language features from VLMs. The aggregated semantic features are embedded into a semantic-aware 3D Gaussian representation, enabling joint reconstruction of appearance, geometry, and semantic information. To handle complex dynamic surgical environments, we further propose semantic-aware deformation tracking (SADT), which models the temporal deformation of both texture and semantic features for more accurate 4D reconstruction. In addition, we introduce a codebook-based query (CQ) module that supports real-time language-assisted 3D segmentation by matching rendered semantic features with text-codebook embeddings. Comprehensive experiments on real-world surgical datasets demonstrate that SurgTPGS++ outperforms state-of-the-art methods in text-promptable 3D surgical scene understanding, highlighting its potential for advancing surgical embodied intelligence. Our code is available at: https://github.com/lastbasket/SurgTPGS.