Skip to content
Log in
#large language models

Retrieval-Augmented Multimodal Large Language Models for Visual Question Answering of Construction Occupational Health and Safety Hazards

Yang Liu Luping Li Xing Su Han Lin Ling Zhang Donghan Guan Hubo Cai
Nov 2026 · Journal of Management in Engineering · 0 citations · 23 references

TL;DR

A visual knowledge enhancement framework for construction OHS visual question answering (VQA) based on multimodal large language models (MLLMs) and retrieval-augmented generation (RAG) is proposed, augmenting managerial capacity for reliable and objective OHS hazard prevention.

Abstract

Construction occupational health and safety (OHS) hazard oversight is a critical pillar of engineering management, requiring the complex integration of dynamic visual evidence with rigorous regulatory standards. Traditional oversight, heavily reliant on manual inspections, is labor intensive and prone to cognitive omissions. While automated hazard detection has evolved, existing paradigms remain constrained by closed-set recognition, failing to simulate the open-ended, heuristic reasoning of safety experts. To bridge this gap, this study proposed a visual knowledge enhancement framework for construction OHS visual question answering (VQA) based on multimodal large language models (MLLMs) and retrieval-augmented generation (RAG). It can reliably respond to site managers’ open questions about OHS hazards in construction images. The method’s primary innovation lies in a tailored RAG framework with a knowledge base for construction OHS hazard VQA, addressing three critical challenges: cross-modal semantic misalignment, knowledge demand variability, and information overload–induced cognitive bias. To enable systematic evaluation of the framework and mitigate the lack of public benchmarks, we designed experiments across three question types and built the Construction Hazard VQA Dataset (ConHazard-VQA), the first dedicated dataset for construction OHS hazard VQA, featuring 1,034 high-quality image-question-answer pairs. Experiments across classification, counting, and open-ended question types confirmed consistent and significant performance gains over the baseline. By introducing a tailored, practical framework that translates a general-purpose MLLM into a domain-specific expert for construction OHS hazard VQA, a new paradigm for data-driven safety management was established. This research advances the body of knowledge in engineering management by transitioning automated oversight from rigid pattern matching to expert-like, diagnostic decision support, thereby augmenting managerial capacity for reliable and objective OHS hazard prevention.

DOI Semantic Scholar
View source

Similar papers

Aug 2026

A large language model-based question-answering system for crack information

This research provides a highly accurate, scalable, and reliable framework for automated bridge defect analysis, offering a practical methodology to enhance data utilization in bridge management.

Luyang Zhang, Xuzhao Lu, Fengzong Gong et al. · 0 citations
Conference Jul 2026

Retrieval-Augmented Large Language Models for Evidence-Based Hazard Log Generation in Emerging Aviation Systems

We introduce a retrieval-augmented synthesis pipeline for deriving structured hazard logs for emerging aviation concepts from historical aviation accident evidence. NTSB accident reports are transformed into a schema-consistent corpus combining coded findings and narrative mechanisms for semantic indexing. Mechanism-level retrieval uses sentence-transformer embeddings, a FAISS inner-product index, evidence-derived seed extraction, and maximal marginal relevance to obtain diversified, scenario-relevant cases. Hazard generation is constrained by strict JSON schema validation, one-to-one evidence binding, explicit causal sequencing, and enforced primary-mechanism uniqueness. A multipass strategy with critic-based filtering and deterministic de-duplication improves robustness against mechanism repetition and evidence drift. Evaluation of an urban eVTOL safety-landing scenario compares locally deployed open-weight models under identical constraints. Retrieval augmentation supports mechanism-specific and traceable hazard derivation compared to unconstrained scenario-based prompting. Mistral-7B requires multipass generation to achieve acceptable mechanism diversity and evidence consistency, whereas GPT-OSS-20b produces structurally valid and mechanism-differentiated hazard sets in a single pass. Scaling to GPT-OSS-120b yields only marginal improvements at substantially higher computational cost.

Michael Schultz, Simon Göppel · 0 citations

A Systematic Evaluation Framework for AI-Driven BIM Question Answering Systems

A systematic evaluation framework for BIM-QA systems comprising a four-category question taxonomy based on data retrieval complexity, and an evaluation protocol with five binary quality criteria, enabling consistent benchmarking and targeted diagnosis of system weaknesses is presented.

Sylvain Hellin, Stavros Nousias, André Borrmann · 2 citations
Review Jul 2026

Large language models for evidence-based planning: Evaluating an SLR-RAG framework for knowledge synthesis of urban vacant land

This study contributes to the field by elucidating the capabilities and limitations of LLMs and RAG in urban studies, highlighting that while text-only retrieval is insufficient for context-rich analysis, future advancements in spatially aware hybrid retrieval offer a promising pathway forward.

Xinyu Wang · 0 citations
2026

Self-Reflective VLM with Multimodal Retrieval-Augmented Generation for Long-Context Construction Regulatory Document Comprehension

This paper presents a self-reflective multimodal RAG-assisted VLM pipeline that augments existing VLMs with an external multimodal RAG mechanism, moving beyond text-only retrieval by jointly leveraging visual representations and semantic summaries to identify relevant evidence pages.

Shuyi Wang, Yuguang Fu, Jinwoo Kim · 0 citations

Related blog posts

Microsoft Research Blog Aug 31, 2026

GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models

What if pathology foundation models could do more with less? GigaPath-Flash and GigaTIME-Flash cut computational demands while maintaining strong performance, opening the door to larger studies and broader exploration. The post GigaPath-Flash and GigaTIME-Flash: Toward population-scale discovery with efficient pathology foundation models appeared first on Microsoft Research.