Skip to content
Book Open access

GraMMA: A Graph-Conditioned Multi-Modal Agent for Fine-Grained Knowledge Reasoning

Nov 2026 · Proceedings of the 1st International Workshop on Agentic Multimodal Intelligence: Models, Benchmarks, and Applications · 0 citations · 9 references

Abstract

Multi-modal Knowledge Graphs (MMKGs) provide a natural substrate for knowledge-intensive question answering because they connect structured relational facts with entity-centered visual evidence. Recent LLM-based reasoning methods have improved multi-hop exploration over knowledge graphs, but extending them to MMKGs remains challenging: agents often observe graph structure only through opaque intermediate states, couple textual and visual evidence too early, and behave unstably in long-horizon search. To address these issues, we propose GraMMA (Graph-Conditioned Multi-Modal Agent), an inspectable framework for MMKG reasoning with three aligned designs: (1) structured subgraph context as explicit observation space, (2) relation-aware cross-modal verification for selective visual grounding, and (3) evolutionary policy refinement for robust exploration. GraMMA routes primarily with textual and structural cues, and invokes visual evidence only when needed through a dedicated CrossModal action. We construct a 300-question evaluation set from ImgFact through human-agent collaboration and evaluate GraMMA on four MMKG benchmarks. Experiments demonstrate that GraMMA substantially outperforms existing methods, particularly in multi-hop reasoning and visual-dependent question answering scenarios.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.