Multimodal Object--Attribute--Relation Graph Construction for Explicit Scene Representation
Abstract
Recent multimodal models have substantially improved scene understanding from heterogeneous inputs such as images, text, and audio. However, rich multimodal description alone is often insufficient for explicit and machine-interpretable scene representation. To address this gap, we introduce Multimodal Object-Attribute-Relation (OAR) Graph Construction, a new task that formulates image, text, and audio inputs describing the same scene as unified structured graph prediction. We propose a modality-aware framework that combines task-specific multimodal fusion, cascaded O-A-R prediction, and learned candidate-pair pruning for efficient graph inference. To facilitate systematic study of this task, we construct VG-Tri, a controlled tri-modal benchmark that extends scene graph annotations with aligned textual and acoustic descriptions, enabling rigorous analysis of how heterogeneous modalities contribute to structured graph prediction. Experiments on VG-Tri show that the proposed method consistently outperforms unimodal baselines, naive multimodal fusion strategies, and extraction-based alternatives, with leakage-controlled analyses confirming that the gains stem from genuine cross-modal semantic integration. The code is available at https://github.com/lcx529955/Multimodal-OAR.