This work proposes SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning and introduces a reinforcement learning strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct.
Abstract
Knowledge-based Visual Question Answering (KB-VQA) aims to answer queries that necessitate reasoning over external knowledge sources beyond the visual content. Typically, current methods fuse multimodal features to retrieve external information, subsequently leveraging Multimodal Large Language Models (MLLMs) to derive answers from the retrieved evidence. However, these methods often struggle to capture structural associations within complex contexts to effectively filter noise. Furthermore, they frequently fail to ensure that the reasoning process remains strictly faithful to the retrieved evidence. To address these challenges, we propose SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning. Specifically, we first employ a coarse-grained hybrid search fusing visual and textual modalities to recall candidate documents, and subsequently implement a structure-aware fine-grained graph retrieval that captures structural dependencies to filter noise and pinpoint precise evidence. Moreover, we introduce a reinforcement learning (RL) strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct. This strict alignment constraint compels the model to anchor its response in the retrieved context, effectively enhancing its capability to locate evidence via multimodal features and perform faithful reasoning. Extensive experiments on the Encyclopedic-VQA and InfoSeek benchmarks demonstrate that SAFE-G outperforms prior methods by a margin of 8.9% and 3.5%, substantially enhancing the overall reasoning accuracy. Our source code is publicly available at: https://github.com/MINE-USTC/SAFE-G.
This work proposes an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA that focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference.
Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we observe that directly concatenating heterogeneous visual descriptions and retrieved knowledge into long, unstructured prompts often degrades reasoning performance, due to both excessive irrelevant context and the lack of explicit relational structure. In this paper, we propose an LLM-based Structured Context Reasoning (SCoRe) framework that infers both explicit and implicit relationships for prediction. SCoRe consists of three stages: Context Acquisition, which generates diverse visual notes and retrieves explicit knowledge via an efficient two-stage multimodal retrieval strategy; Context Selection, which filters relevant visual, explicit, and implicit knowledge using LLM-guided selection; and Context Compression, which performs Relational Logic Distillation (RLD) to transform raw text into explicit entity-relation triplets. These relational triplets serve as a concise and structured prompt for final answer prediction. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SCoRe consistently outperforms state-of-the-art methods.
Qiyou Liu, Yong Zhang, Jianjie Luo et al.· 0 citations
Text-based visual question answering (TextVQA) requires reasoning over images containing rich textual content, often involving knowledge beyond what is directly observable. Existing methods fuse visual objects and OCR tokens but struggle when questions require external knowledge. Moreover, naively incorporating retrieved knowledge often introduces irrelevant or misleading information, which may hinder reasoning rather than support it. To address these challenges, we propose a TextVQA framework that integrates external prior knowledge to support multimodal reasoning. Given an image and question, a task-aware knowledge retrieval module selects relevant candidates, which are then filtered and verified by a knowledge verification module leveraging large language models. The verified knowledge and question are compressed into compact embeddings via a perceiver-based semantic resampler and jointly processed with visual and OCR features in a multimodal reasoning module. Experiments on the TextVQA and ST-VQA datasets demonstrate that our approach effectively leverages external knowledge to improve performance on knowledge-intensive questions.
Dynamic Multi-Path Retrieval for KB-VQA (DMRAG) is proposed, which re-trieves candidates through multiple retrieval paths that capture complementary visual and semantic cues and performs Question-Adaptive Gated Fusion to balance contributions from different modalities according to the query’s information need.
Zeyu Song, Yimin Deng, Yu-Xin Zhang et al.· Proceedings of the Thirty-Fi...· 0 citations
The results show that structured evidence planning and compact graph-guided retrieval can support competitive multimodal question answering without target-dataset fine-tuning or iterative retrieval.
Zafar Ali, Asad Khan, Nimbeshaho Thierry et al.· 0 citations
KBMR is proposed, the first MLLM-based embedding retriever tailored for KB-VQA, and an MLLM-based semantic discriminator that generates continuous entity-consistency weights is introduced to tackle the challenge of noisy supervision in Wikipedia-scale retrieval.
Hangrui Xu, Zhengxian Wu, Yu Yu et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
