GraMMA: A Graph-Conditioned Multi-Modal Agent for Fine-Grained Knowledge Reasoning
Abstract
Multi-modal Knowledge Graphs (MMKGs) provide a natural substrate for knowledge-intensive question answering because they connect structured relational facts with entity-centered visual evidence. Recent LLM-based reasoning methods have improved multi-hop exploration over knowledge graphs, but extending them to MMKGs remains challenging: agents often observe graph structure only through opaque intermediate states, couple textual and visual evidence too early, and behave unstably in long-horizon search. To address these issues, we propose GraMMA (Graph-Conditioned Multi-Modal Agent), an inspectable framework for MMKG reasoning with three aligned designs: (1) structured subgraph context as explicit observation space, (2) relation-aware cross-modal verification for selective visual grounding, and (3) evolutionary policy refinement for robust exploration. GraMMA routes primarily with textual and structural cues, and invokes visual evidence only when needed through a dedicated CrossModal action. We construct a 300-question evaluation set from ImgFact through human-agent collaboration and evaluate GraMMA on four MMKG benchmarks. Experiments demonstrate that GraMMA substantially outperforms existing methods, particularly in multi-hop reasoning and visual-dependent question answering scenarios.