This work proposes an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA that focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference.
Abstract
Knowledge-based visual question answering requires models to connect question-relevant visual evidence with external knowledge for accurate reasoning. However, existing approaches still face two critical challenges: insufficient alignment between visual content and question intent, which leads to missing or irrelevant evidence, and ambiguity in entity-level semantics, especially for fine-grained or knowledge-intensive concepts. To address these issues, we propose an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA. The framework uses lightweight vision–language models to construct structured textual evidence, which is subsequently provided to a frozen large language model (LLM) for final reasoning. We emphasize that the proposed method does not introduce a new LLM architecture or a new training mechanism; instead, it focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference. The framework consists of four complementary evidence-construction modules: (1) a question-oriented image information extraction module that generates query-relevant visual descriptions to enhance visual–semantic alignment; (2) an entity enhancement module that introduces clarifying sub-questions to alleviate entity-level ambiguity; (3) a candidate-guided answer generation module that provides plausible answer cues to constrain the reasoning space; and (4) a contextual exemplar retrieval module that supplies relevant demonstrations to support knowledge-grounded inference. Experiments on OK-VQA and A-OKVQA show that the proposed method achieves 66.72% and 69.51% accuracy, respectively, outperforming strong baselines, while supplementary analyses examine its robustness, output-format reliability, and inference cost.
This work proposes SAFE-G, a Structure-Aware Faithful Evidence-guided Generation framework, which enables precise evidence localization and trustworthy reasoning and introduces a reinforcement learning strategy with an evidence-grounded reward that assigns credit to correct answers only when the selected evidence is correct.
Long Shu, Shuochen Liu, Wei Chen et al.· 0 citations
Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we observe that directly concatenating heterogeneous visual descriptions and retrieved knowledge into long, unstructured prompts often degrades reasoning performance, due to both excessive irrelevant context and the lack of explicit relational structure. In this paper, we propose an LLM-based Structured Context Reasoning (SCoRe) framework that infers both explicit and implicit relationships for prediction. SCoRe consists of three stages: Context Acquisition, which generates diverse visual notes and retrieves explicit knowledge via an efficient two-stage multimodal retrieval strategy; Context Selection, which filters relevant visual, explicit, and implicit knowledge using LLM-guided selection; and Context Compression, which performs Relational Logic Distillation (RLD) to transform raw text into explicit entity-relation triplets. These relational triplets serve as a concise and structured prompt for final answer prediction. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SCoRe consistently outperforms state-of-the-art methods.
Qiyou Liu, Yong Zhang, Jianjie Luo et al.· 0 citations
Text-based visual question answering (TextVQA) requires reasoning over images containing rich textual content, often involving knowledge beyond what is directly observable. Existing methods fuse visual objects and OCR tokens but struggle when questions require external knowledge. Moreover, naively incorporating retrieved knowledge often introduces irrelevant or misleading information, which may hinder reasoning rather than support it. To address these challenges, we propose a TextVQA framework that integrates external prior knowledge to support multimodal reasoning. Given an image and question, a task-aware knowledge retrieval module selects relevant candidates, which are then filtered and verified by a knowledge verification module leveraging large language models. The verified knowledge and question are compressed into compact embeddings via a perceiver-based semantic resampler and jointly processed with visual and OCR features in a multimodal reasoning module. Experiments on the TextVQA and ST-VQA datasets demonstrate that our approach effectively leverages external knowledge to improve performance on knowledge-intensive questions.
Dynamic Multi-Path Retrieval for KB-VQA (DMRAG) is proposed, which re-trieves candidates through multiple retrieval paths that capture complementary visual and semantic cues and performs Question-Adaptive Gated Fusion to balance contributions from different modalities according to the query’s information need.
Zeyu Song, Yimin Deng, Yu-Xin Zhang et al.· Proceedings of the Thirty-Fi...· 0 citations
EKS is a novel framework that leverages entity relations in commonsense knowledge graphs to dynamically generate knowledge sentences relevant to both visual and textual entities and formulates knowledge selection as a relevance scoring problem, where semantic similarity is used to measure the relevance between knowledge candidates and the input question.
Kun Zhu, Kun Zhou, De-Xin Zhao· Multimedia Systems· 0 citations
Visual Question Answering (VQA) aims to achieve cross-modal semantic understanding through joint modeling of visual content and natural language. Although existing attention-based approaches effectively align features, they struggle to simultaneously accommodate global semantic modeling and local fine-grained perception within single-path architectures, leading to semantic drift and underutilization of information when addressing multi-perspective reasoning tasks. To mitigate this limitation, we propose a Decoupled Global-Local Collaborative Network (DGL-Net) that structurally partitions the cross-modal reasoning process into two complementary branches: global semantic understanding and local fine-grained modeling. Specifically, the global branch employs hierarchical attention fusion to model long-range dependencies and holistic scene semantics, providing contextual constraints for reasoning. The local branch introduces semantic relation enhancement and relevant region screening to explicitly model intra-modal structural relationships, strengthening fine-grained region-word alignment by injecting structural information. During training and inference, a branch-decoupling and collaborative fusion strategy enables independent yet complementary feature learning. Extensive experimental results demonstrate accuracies of 72.15% and 57.87% on VQA-v2 and GQA benchmarks, respectively. Notably, the proposed method exhibits enhanced robustness and generalization in complex semantic and compositional reasoning tasks, validating the efficacy of the decoupled collaborative strategy.
Gan-Long Zhou, Dezhi Han, Xiang Shen et al.· Computer Science and Informa...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
