SemanticAdapter: parameter-efficient adaptation for label-efficient 3D point cloud semantic segmentation
TL;DR
SemanticAdapter is proposed, a parameter-efficient adaptation method that freezes the pretrained 3D encoder and introduces two lightweight components: a Cross-Modal Attention (CMA) module using point cloud features as queries and CLIP text embeddings as keys/values for explicit semantic alignment.
Abstract
CLIP2Scene transfers CLIP's semantic knowledge to 3D point cloud networks via cross-modal contrastive learning, achieving annotation-free 3D semantic segmentation. However, its inference relies solely on cosine similarity for classification, failing to fully exploit the semantic alignment potential. We propose SemanticAdapter, a parameter-efficient adaptation method that freezes the pretrained 3D encoder and introduces two lightweight components: (1) a Cross-Modal Attention (CMA) module using point cloud features as queries and CLIP text embeddings as keys/values for explicit semantic alignment; and (2) a zero-parameter Hierarchical Classifier (HC) leveraging coarse-to-fine category mappings. Experiments show that SemanticAdapter improves mIoU from 18.2% to 22.3% on nuScenes under annotation-free settings, and achieves 60.0% mIoU with 1% labels on nuScenes and 62.3% mIoU with 10% labels on SemanticKITTI, outperforming or matching state-of-the-art methods.