Jul 2026· Automation in Construction· Vol 190, pp. 107117· 0 citations· 57 references
Computer Science
TL;DR
Building-MLLM is proposed, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.
Abstract
Point cloud-based understanding has become an important enabler for facility operation and maintenance involving indoor building components. However, existing methods output only discrete labels without explaining component functions or natural language interactions. This paper proposes Building-MLLM, a point cloud-centered multimodal large language model (MLLM) for indoor components, which models point clouds and instructions to generate responses across Simple Recognition, Complex Captioning, and Multi-Engineering Question Answering tasks. Building-MLLM addresses semantic concentration through four domain-specific mechanisms: Point Information Enhancer for task-relevant semantics, Geometry-Preserving Regularization preventing geometric erosion, fixed textual prefix for domain stabilization, and multi-dimensional LoRA balancing recognition with reasoning. A multi-constraint progressive instruction-generation engine is developed to compile a synthetic point cloud-text dataset with 4198 objects, 37,782 instruction-following pairs, and 47 categories. Experiments show that Building-MLLM achieves 88.00%, 65.10%, and 68.14% on the three task types, respectively, demonstrating superior indoor component language understanding and providing initial generalizability in transfer inference on other real-world datasets.
Vision-language models (VLMs) achieve strong semantic understanding but remain unreliable in metric spatial reasoning, particularly when queries require comparing multiple instances of the same object category. We study this problem through the Closest-Instance Distance Query (CIDQ), where a model must identify the nearest visible candidate to a unique reference object and estimate their gravity-aligned floor-plane distance. We introduce SPATIALQUERY, a training- free framework for CIDQ reasoning from a single RGB image, together with SPATIALQUERY-1M, a benchmark containing over one million RGB-only question-answer pairs from 200 indoor scenes. SPATIALQUERY recovers instance-level metric geometry and transforms it into a canonical Bird's-Eye View through Scene Cubifying, which represents objects as uniformly sized, category-coded blocks to emphasize their relative floor- plane locations. We further propose Uncertainty-Aware Chain-of-Thought (UA-CoT) prompting, which incorporates geometry- derived per-instance uncertainty into the VLM reasoning process. Without task-specific fine-tuning or architectural modification, SPATIALQUERY with Qwen3-VL-8B achieves a Floor-MAE of 0.259 m, an Unc-Acc@0.3 m of 90.5%, and a proximity-decision accuracy of 84.18%, outperforming fine-tuned spatial specialists, general-purpose VLMs, and closed-source frontier models. Code, benchmark resources, and an interactive demo are available at https://namhai1810.github.io/SpatialQuery/.
The development of 3D large multimodal models (3D-LMMs) has advanced research on 3D visual question answering. Yet most existing methods rely on implicit feature mapping, where point clouds or scene features are directly projected into the latent space of a language model, without explicitly modeling local spatial structure. In complex indoor environments, this design can lead to spatial judgments that are inconsistent with the actual physical layout during 3D visual question answering, a phenomenon referred to as spatial hallucination. To address this issue, this paper proposes TA-LMM, a 3D visual question answering method built on explicit scene graphs and local topological priors. The method begins by parsing a raw 3D scene into an explicit spatial-semantic scene graph and extracting instance-level representations that encode both semantic features and geometric location information. It then constructs a local physical neighborhood around the target object, serializes neighboring objects together with their distance information into structured priors, and injects them into the multimodal reasoning sequence as conditional context. Under the current experimental setting on the Replica dataset, the results show that the proposed method achieves strong performance on spatial-relation question answering while also alleviating spatial hallucination to a certain extent. These findings suggest that explicit local topological priors can improve scene consistency in 3D visual question answering.
Kaixin Wu, Kunlin Zhou, Boxin Li et al.· International Conference on...· 0 citations
3D point cloud understanding, e.g. out-of-distribution detection and recognition, remains a challenge, particularly in applications where safe and robust perception is critical. While recent 3D VLMs enable direct recognition via text prompts, their predictions are often unstable when the test distribution shifts or when only a handful of labeled samples are available. In this paper, we present GSP++, a graph-based inference framework that exploits the manifold structure of test-time point clouds to refine VLM scores without additional training. GSP++ constructs a graph over class prototypes, optional few-shot samples, and unlabeled test instances, and performs log-scaled score propagation to improve robustness across different dataset sizes while keeping computation efficient. The framework enables unified 3D OOD detection and classification. Moreover, we introduce a self-training strategy that selects high-confidence positive and negative samples and assigns them calibrated pseudo scores to further stabilize propagation. To enable practical deployment, we introduce dynamic graph construction and maintenance strategies that efficiently manage computation and memory, enabling scalable 3D understanding under both transductive and incremental inference. Extensive experiments on six datasets show that GSP++ not only reproduces and strengthens the full suite of OOD detection evaluations from GSP, a prior conference version, but also consistently improves zero-shot and few-shot 3D classification over strong VLM baselines. The code is available on https://github.com/handsome999KK/GSP.
Tiankai Chen, Yushu Li, Adam Goodge et al.· IEEE Transactions on Pattern...· 0 citations
Semantic 3D city models provide rich geometric and semantic information, but remain challenging for non-experts and interdisciplinary researchers to access and query due to their complex structures and specialized data formats. To address this issue, we present CityLLM, a framework for natural-language querying of semantic 3D city models alongside complementary urban datasets. The framework combines spatial and graph databases within an LLM-based workflow that supports iterative query refinement and cross-database chaining. We evaluate CityLLM on a CityJSON dataset of Rotterdam (853 LoD2 buildings) using GPT-OSS, Gemini 3.1, and GPT-5.4, along with selected variants, across multiple metrics: answer correctness, visualization correctness, query success, and retry attempts. A total of 54 natural-language queries are curated across four scenarios: spatial, graph, cross-database, and conversational. Results show strong overall performance, with answer correctness ranging from 85.2% to 100%, visualization correctness from 92.9% to 100%, a 100% query success rate, and fewer than three retries across all 54 queries. Overall, the findings suggest that CityLLM provides a lightweight and extensible approach for conversational access to semantic 3D city data.
Rabindra Lamsal, S. Zlatanova, John Shen· 0 citations
Visual Language Navigation (VLN) aims to enable an embodied agent to navigate complex environments by following natural language instructions. Recent approaches build semantic spatial maps and leverage Large Language Models (LLMs) for reasoning and decision making. Despite these advances, existing systems lack instance-level object detail and robustness to diverse user queries, limiting reliable navigation in complex indoor environments. To address these limitations, we propose Instance-Enriched Semantic Maps, a unified framework with three key contributions: (1) Instance-level two-and-a-half-dimensional (2.5D) rich information mapping that constructs maps from color and depth observations via open-vocabulary panoptic segmentation, preserving vertical distinctions and capturing small objects, while storing diverse semantic attributes and natural language captions enriched with room-level context. (2) Robust query processing via LLM-based target selection, which dynamically routes queries across type-specialized experts and integrates their outputs through score-level fusion, enabling consistent goal selection across diverse query formulations. (3) Storage-efficient semantic representation that achieves approximately 96% reduction compared to three-dimensional (3D) scene-graph approaches while preserving sufficient spatial information for navigation. The proposed 2.5D representation outperforms the 3D baseline by over 27% in prediction-normalized Area Under the Curve (AUC). In navigation experiments, our method achieves over 17% improvement in object retrieval and over 23% in navigation success compared to the baseline across diverse query types. The project page is available at https://rcilab.github.io/iesm_vln.
J. Hong, Eunae Kang, Sanghyun Kim et al.· Engineering applications of...· 0 citations
Open-vocabulary 3D scene understanding aims to segment 3D scenes beyond predefined categories by transferring semantic knowledge from vision-language models. Existing methods have advanced this task by lifting language-aligned 2D features into 3D, yet they often rely on context-independent semantic representations, leaving object relationships underexplored for contextual refinement. We propose RelGraphOV, a relationship-aware framework that uses 3D scene graphs to enhance open-vocabulary 3D understanding. Our method constructs relational scene graphs from multi-view observations by leveraging vision-language reasoning to infer object relationships and prune geometrically implausible connections, without manual relationship annotations. To aggregate relational context while avoiding feature interference, we introduce an Adaptive Gated Dual-Stream Contextual GAT that separates dense geometric features and semantic CLIP embeddings, performs edge-guided message passing, and adaptively fuses complementary semantics. A hierarchical contrastive objective further promotes instance-level consistency and category-level discrimination. Experiments on ScanNetV2, ScanNet200, ScanNet$++$, and Replica demonstrate strong performance and generalization ability. Project Page: https://cxavireh.github.io/relgraphov-projectpage
Xianhao Chen, Jiarui Hu, Yuanbo Yang et al.· 0 citations