When Do Multimodal and Graph-Augmented RAG Help? A Controlled Evaluation for Document Question Answering

Abstract

Retrieval-augmented generation (RAG) systems commonly operate on text extracted from documents, potentially losing information contained in figures, tables, layout, and relationships distributed across passages. We present an explainable multimodal graph-RAG architecture that augments a text-only baseline with LLM-extracted subject--relation--object triples and CLIP-based retrieval of figures and tables. The three evidence sources are retrieved independently and fused only at generation time, allowing the effects of graph evidence, visual evidence, and generator choice to be evaluated separately. We conduct a controlled four-way ablation on 1,000 PubLayNet pages across single-passage, multi-hop, and figure questions using two closed-weight and two open-weight multimodal generators. We also compare matched caption-answerable and pixel-only figure question sets to distinguish caption recovery from genuine visual question answering. Knowledge-graph augmentation provides no reliable accuracy improvement across generators or question types in this corpus. On pixel-only questions, text-only systems achieve zero accuracy, while multimodal systems reach 0.057--0.114 and are constrained by both image retrieval (Recall@3 = 0.371) and the generators'ability to interpret dense scientific figures and tables. Caption-derived questions substantially overestimate text-only visual question-answering capability. Processing the same image also produces an elevenfold variation in input tokens across generators, showing that image tokenisation can dominate deployment cost. Source code, experiment configurations, and evaluation resources are publicly available at https://github.com/sokistar24/multimodal-graph-rag. These findings show that the value of multimodal and graph augmentation depends on retrieval design, corpus structure, benchmark construction, and generator efficiency.