LieVLM is introduced, a Lie routing framework for view-robust 3D understanding that retains the pretrained semantic and direct spatial context while learning view-dependent feature adjustments.
Abstract
Multimodal Large Language Models (MLLMs) have made substantial progress in 3D scene understanding from multi-view RGB observations. Yet a fixed object and query can elicit inconsistent target-related responses across camera viewpoints, a phenomenon we characterize as viewpoint-induced semantic drift. We introduce LieVLM, a Lie routing framework for view-robust 3D understanding. Motivated by the rotational component of viewpoint change, Lie routing adapts visual representations by applying local $\mathrm{SO}(3)$ transformations to projected features within learned three-dimensional subspaces. Geometry tokens extracted by VGGT predict local rotation parameters and channel-wise routing coefficients. A closed-form Pad\'e--Cayley map transforms projected visual features, whose identity-relative corrections are projected back and gated into the original visual and geometry features. This design retains the pretrained semantic and direct spatial context while learning view-dependent feature adjustments. Experiments on 3D captioning, grounding, detection, and spatial reasoning show consistent improvements over additive fusion across training scales and backbones. Controlled ablations examine the roles of geometric conditioning and transformation structure. Viewpoint analyses further reveal systematic variation in the predicted routing states alongside more localized target responses and stronger task performance across viewpoint changes.
This work proposes \textbf{novel-view semantic rendering} as an auxiliary training task that requires the model to predict the semantic layout of an unobserved viewpoint, inspired by humans'ability to mentally simulate novel viewpoints during spatial reasoning.
Yuqun Wu, Yao Xiao, Chu-Hang Zou et al.· 0 citations
Monocular 3D object detection offers a low-cost route to 3D perception, but it remains fundamentally ill-posed, since depth must be inferred from a single view. Recent transformer-based detectors learn geometry purely from image features and 3D box annotations, lacking the high-level semantic priors that vision-languag...
K. Vu, Cao Le Minh Khoa, Nguyen Viet Tri Pham et al.· 0 citations
Unified 3D vision-language systems must combine complementary geometry, scale, and appearance cues while supporting tasks from instance segmentation to language-guided reasoning. Existing methods often process point clouds, voxel grids, and multi-view images independently; directly combining these heterogeneous represe...
Xue-Qi Qiu, Xing-Yu Miao, Jing-Jing Deng et al.· arXiv.org· 0 citations
In multi-modal 3D pre-training, aligning 3D shapes, their 2D image counterparts, and language descriptions has made tremendous progress. However, we find that supervision from rendered images is unreliable, the semantics encoded in synthetic views are distorted due to domain gap, self-occlusion, and rendering artifacts...
Chao-Yang Zhu, Zhe Liao, Jia-Yi Zhang et al.· 0 citations
This work proposes a segmentation framework guided by Contrastive Language–Image Pre-training (CLIP) that enriches sparse 3D tokens with vision–language semantic priors and introduces a decoupled CLIP-induced semantic residual that forms semantic-geometric attention biases for local window attention.
3D vision-language understanding is central to multimedia research. Representative tasks such as visual grounding and text-to-shape retrieval require aligning 3D geometry with natural language that often depends on fine-grained geometric cues. Most existing 3D-language models build on generic backbones with limited geo...
Yi-Long Guo, Jin-Hui Wu, Hong-Zhe Liu et al.· Proceedings of the 34th ACM...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.