Skip to content

GOMA: Toward Structure-Driven Multimodal Alignment from a Graph Signal Smoothing Perspective

May 2026 · arXiv.org · Vol abs/2605.15723 · 0 citations · 64 references
Computer Science

TL;DR

Graph-Optimized Multimodal Alignment (GOMA), which lets a jointly trained model support single-modality and dual-attribute retrieval through a task-specific readout, achieves state-of-the-art performance on all 14 primary measures against 14 external methods.

Abstract

Multimodal retrieval uses images, text, and object relationships to answer different questions about the same collection. A query may seek an object's paired description, another object in the same category, or an object connected by an observed relationship. These goals rely on different notions of relevance. Paired matching requires object-specific distinctions, whereas cross-object retrieval benefits from relational agreement. Existing methods learn strong cross-modal correspondence or propagate information over a graph, but a shared output regularized toward neighbors can weaken identity distinctions. Moreover, gains from graph regularization can diminish after uniform graph propagation. We introduce Graph-Optimized Multimodal Alignment (GOMA), which assigns these roles to two connected embeddings. Each modality produces a content embedding directly supervised for paired identity and a semantic embedding jointly trained with cross-modal pairs and observed relationships. For complete records, both embeddings form an initial fused representation, semantic agreement sets positive weights on observed edges, and restart graph propagation reinjects this initial signal. This design lets a jointly trained model support single-modality and dual-attribute retrieval through a task-specific readout. Across six datasets and four tasks, GOMA achieves state-of-the-art performance on all 14 primary measures against 14 external methods. Controlled comparisons further show how separate supervision, graph regularization, and semantic-guided graph propagation shape the final representation and align the learned signal with each retrieval target.

View source

Similar papers

Book Open access Nov 2026

GraphMatch: A Dual-Channel Framework for Compositional Cross-Modal Retrieval

Cross-modal retrieval has made remarkable strides with embedding-based methods that map different modalities into a unified space. However, these approaches fail on compositional queries that require understanding multiple entities and their relationships. The root cause is the flat feature paradigm, which compresses s...

Xin-Biao Gan, Hong-Xu Jin · 0 citations
Preprint Nov 2026

Fuse To Align: Multimodal Representation Learning for Retrieval via Hypergraph Regularization

Multimodal retrieval must integrate complementary evidence from views that may be missing, noisy, or unequally informative. When a sample provides limited reliable evidence, learning from its views alone can hinder the acquisition of useful retrieval representations. Semantically related training samples may provide co...

Anindya Nag, Ambuj Mehrish, Sebastiano Vascon · 0 citations
#machine learning Preprint Oct 2026

Toward Omni Multimodal Graph Foundation Model: A Topology-Driven Binding Approach

GraphBind is a topology-driven approach that uses graph topology to bind rich modality information into a unified shared space that integrates structure and semantics, and adapts this space to discriminative and generative tasks through lightweight interfaces.

Xun-Kai Li, Chen-Xi Wan, Yin-Lin Zhu et al. · 0 citations
Open access Nov 2026

Complex-Valued Relational Embeddings With Amplitude-Phase Decomposition for Scene Graph Generation.

Learning compact and discriminative representations of object relations is central to structured visual reasoning. In scene graph generation, these relations must be inferred from ambiguous visual cues under a heavy class imbalance; yet most methods encode a relation as a single real-valued vector that entangles intera...

Namgyu Jung, Chang Choi · 0 citations
Book Open access Nov 2026

Multimodal Object--Attribute--Relation Graph Construction for Explicit Scene Representation

Recent multimodal models have substantially improved scene understanding from heterogeneous inputs such as images, text, and audio. However, rich multimodal description alone is often insufficient for explicit and machine-interpretable scene representation. To address this gap, we introduce Multimodal Object-Attribute-...

Chen-Xing Li, Yi-Ping Duan, Xiao-Ming Tao et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Oct 28, 2026

Making machines that can listen

Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.

Microsoft Research Blog Aug 11, 2026

Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement

Radiology AI is evolving beyond report generation. CARE-X explores a unified approach that combines flexible reasoning, calibrated predictions, and measurement-based tools for chest X-ray interpretation. The post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.