May 2026· arXiv.org· Vol abs/2605.15723· 0 citations· 64 references
Computer Science
TL;DR
Graph-Optimized Multimodal Alignment (GOMA), which lets a jointly trained model support single-modality and dual-attribute retrieval through a task-specific readout, achieves state-of-the-art performance on all 14 primary measures against 14 external methods.
Abstract
Multimodal retrieval uses images, text, and object relationships to answer different questions about the same collection. A query may seek an object's paired description, another object in the same category, or an object connected by an observed relationship. These goals rely on different notions of relevance. Paired matching requires object-specific distinctions, whereas cross-object retrieval benefits from relational agreement. Existing methods learn strong cross-modal correspondence or propagate information over a graph, but a shared output regularized toward neighbors can weaken identity distinctions. Moreover, gains from graph regularization can diminish after uniform graph propagation. We introduce Graph-Optimized Multimodal Alignment (GOMA), which assigns these roles to two connected embeddings. Each modality produces a content embedding directly supervised for paired identity and a semantic embedding jointly trained with cross-modal pairs and observed relationships. For complete records, both embeddings form an initial fused representation, semantic agreement sets positive weights on observed edges, and restart graph propagation reinjects this initial signal. This design lets a jointly trained model support single-modality and dual-attribute retrieval through a task-specific readout. Across six datasets and four tasks, GOMA achieves state-of-the-art performance on all 14 primary measures against 14 external methods. Controlled comparisons further show how separate supervision, graph regularization, and semantic-guided graph propagation shape the final representation and align the learned signal with each retrieval target.
Cross-modal retrieval has made remarkable strides with embedding-based methods that map different modalities into a unified space. However, these approaches fail on compositional queries that require understanding multiple entities and their relationships. The root cause is the flat feature paradigm, which compresses s...
Xin-Biao Gan, Hong-Xu Jin· Proceedings of the 34th ACM...· 0 citations
Multimodal retrieval must integrate complementary evidence from views that may be missing, noisy, or unequally informative. When a sample provides limited reliable evidence, learning from its views alone can hinder the acquisition of useful retrieval representations. Semantically related training samples may provide co...
This work introduces ICE (Interaction-aware Clifford Encoder), a multimodal graph foundation model built on a node-indexed Clifford latent field that ranks first in all 30 reported supervised and few-shot comparisons.
GraphBind is a topology-driven approach that uses graph topology to bind rich modality information into a unified shared space that integrates structure and semantics, and adapts this space to discriminative and generative tasks through lightweight interfaces.
Xun-Kai Li, Chen-Xi Wan, Yin-Lin Zhu et al.· 0 citations
Learning compact and discriminative representations of object relations is central to structured visual reasoning. In scene graph generation, these relations must be inferred from ambiguous visual cues under a heavy class imbalance; yet most methods encode a relation as a single real-valued vector that entangles intera...
Namgyu Jung, Chang Choi· IEEE Transactions on Neural...· 0 citations
Recent multimodal models have substantially improved scene understanding from heterogeneous inputs such as images, text, and audio. However, rich multimodal description alone is often insufficient for explicit and machine-interpretable scene representation. To address this gap, we introduce Multimodal Object-Attribute-...
Chen-Xing Li, Yi-Ping Duan, Xiao-Ming Tao et al.· Proceedings of the 34th ACM...· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduOct 28, 2026
Professor Paris Smaragdis develops computers with superhuman hearing for applications like speech recognition, music understanding, and environmental monitoring.
Exploring how generative AI could make machine vision more accessible to businesses. The post GenEye in a Box: Making Machine Vision Something You Can Just Ask For appeared first on GPT-Lab.
Microsoft Research Blog· microsoft.comAug 11, 2026
Radiology AI is evolving beyond report generation. CARE-X explores a unified approach that combines flexible reasoning, calibrated predictions, and measurement-based tools for chest X-ray interpretation. The post Introducing CARE-X: Towards Clinically Useful Radiology VLMs with Auxiliary Supervision, Reward-Aligned Learning, and Tool-Augmented Measurement appeared first on Microsoft Research.