Skip to content

GoDeep: Annotation-Free Open-Vocabulary 3D Scene Understanding via Language-Space Lifting

Sep 2026 · arXiv.org · Vol abs/2609.09082 · 0 citations · 37 references
Computer Science

TL;DR

This work uses a vision-language model purely as a translator to produce structured, entity-level descriptions of each posed image, and proposes an aggregation strategy, as a proof of concept, that favors finer elements localization in language-space embeddings.

Abstract

Open vocabulary 3D semantic segmentation methods typically lift CLIP features into 3D. This embeds points in a joint vision-language space known to behave like a bag-of-words on compositional tasks. Furthermore, even annotation free variants often require a large 3D training corpus and a dedicated 3D encoder per domain. Instead we use a vision-language model purely as a translator. It produces structured, entity-level descriptions of each posed image. These descriptions are grounded, projected, and aggregated directly in a general-purpose, language-only embedding space, with no 3D training corpus or encoder required. On ScanNet++, our pipeline is competitive with strong annotation free baselines trained on ScanNet. On a 5-building cultural heritage benchmark, raw scores initially favor a CLIP-based variant, but a single systematic vocabulary correction reverses this ranking. An effect confirmed by a second, independent correction on a different class, indicating that language-space embeddings track physical content more faithfully. This fidelity extends to genuinely out-of-vocabulary (OOV) objects on ScanNet++ proving that language-space embeddings separate presence from absence objects far more sharply than CLIP-based embeddings do. GoDeep also localize these OOV objects within the scene, all without any 2D-3D annotation. Because every representation remains discrete text, predictions are also explainable at the point level. Finally, exploiting both a heuristic weighting, that favors precise over merely frequent observations and GoDeep's explainability property, we propose an aggregation strategy, as a proof of concept, that favors finer elements localization.

View source

Similar papers

Preprint Oct 2026

Lang3DSeg: Annotation-Free Open-Vocabulary 3D Segmentation with Point Transformers

Accurate 3D semantic perception is critical for safe autonomous navigation. However, supervised LiDAR segmentation remains tied to closed taxonomies and to the cost of point-wise manual annotation. Open-vocabulary methods avoid that cost by projecting the output of 2D vision-language models onto LiDAR and distilling it...

Cigdem Kokenoz, A. Salarpour, Alkim Domeke et al. · 0 citations
Preprint Oct 2026

Query-Time Calibration of CLIP Features for Open-Vocabulary 3D Segmentation

Lifting vision-language features such as CLIP into 3D Gaussian reconstructions enables open-vocabulary segmentation through natural-language queries. Although CLIP represents a broad range of semantic concepts, the lifted features of individual scenes can exhibit strong directional concentration, while raw cosine simil...

D. Engel, Yousef Rajeh, Leon Sick et al. · 1 citation
Sep 2026

TRACKGRAPH: Online Open-Vocabulary 3D Scene Graphs via Image-Space Tracking

Open-vocabulary 3D maps enable robots to reason about previously unknown environments using natural language. However, existing systems typically segment every incoming image, associate detections with persistent 3D segments, and frequently perform costly Vision-Language (VL) inference. We present TRACKGRAPH, an online...

Peder Borge Hellesylt, Albert Gassol Puigjaner, Kostas Alexis et al. · 0 citations
Book Open access Nov 2026

E3CoverNet: Progressive E(3)-Aware 3D Backbone for Geometry-Sensitive Vision-Language Alignment

3D vision-language understanding is central to multimedia research. Representative tasks such as visual grounding and text-to-shape retrieval require aligning 3D geometry with natural language that often depends on fine-grained geometric cues. Most existing 3D-language models build on generic backbones with limited geo...

Yi-Long Guo, Jin-Hui Wu, Hong-Zhe Liu et al. · 0 citations
Preprint Oct 2026

OVM-Occ: Open-Vocabulary Monocular Occupancy Prediction via Geometry-Aware Vision-Language Distillation

Monocular occupancy prediction in indoor scenes aims to recover a scene's 3D geometry and semantics from a single RGB image. Existing methods are restricted to a closed set of predefined categories, limiting their ability to generalize to unseen categories. This limitation is particularly critical in indoor scenes, whe...

Shu-Yang Yu, Hai-Bo Qin, Shao-Han Hu et al. · 0 citations
#artificial intelligence Preprint Oct 2026

Revisiting Triplet Contrastive Learning for 3D Shape Understanding

In multi-modal 3D pre-training, aligning 3D shapes, their 2D image counterparts, and language descriptions has made tremendous progress. However, we find that supervision from rendered images is unreliable, the semantics encoded in synthetic views are distorted due to domain gap, self-occlusion, and rendering artifacts...

Chao-Yang Zhu, Zhe Liao, Jia-Yi Zhang et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.