TL;DR
EKS is a novel framework that leverages entity relations in commonsense knowledge graphs to dynamically generate knowledge sentences relevant to both visual and textual entities and formulates knowledge selection as a relevance scoring problem, where semantic similarity is used to measure the relevance between knowledge candidates and the input question.
Text-based visual question answering (TextVQA) requires reasoning over images containing rich textual content, often involving knowledge beyond what is directly observable. Existing methods fuse visual objects and OCR tokens but struggle when questions require external knowledge. Moreover, naively incorporating retrieved knowledge often introduces irrelevant or misleading information, which may hinder reasoning rather than support it. To address these challenges, we propose a TextVQA framework that integrates external prior knowledge to support multimodal reasoning. Given an image and question, a task-aware knowledge retrieval module selects relevant candidates, which are then filtered and verified by a knowledge verification module leveraging large language models. The verified knowledge and question are compressed into compact embeddings via a perceiver-based semantic resampler and jointly processed with visual and OCR features in a multimodal reasoning module. Experiments on the TextVQA and ST-VQA datasets demonstrate that our approach effectively leverages external knowledge to improve performance on knowledge-intensive questions.
Dynamic Multi-Path Retrieval for KB-VQA (DMRAG) is proposed, which re-trieves candidates through multiple retrieval paths that capture complementary visual and semantic cues and performs Question-Adaptive Gated Fusion to balance contributions from different modalities according to the query’s information need.
Knowledge-based Visual Question Answering aims to answer questions about an image by integrating external knowledge with visual and textual information. Recent approaches often rely on in-context learning to prompt Large Language Models (LLMs) with multimodal context in a zero-shot or few-shot manner. However, we observe that directly concatenating heterogeneous visual descriptions and retrieved knowledge into long, unstructured prompts often degrades reasoning performance, due to both excessive irrelevant context and the lack of explicit relational structure. In this paper, we propose an LLM-based Structured Context Reasoning (SCoRe) framework that infers both explicit and implicit relationships for prediction. SCoRe consists of three stages: Context Acquisition, which generates diverse visual notes and retrieves explicit knowledge via an efficient two-stage multimodal retrieval strategy; Context Selection, which filters relevant visual, explicit, and implicit knowledge using LLM-guided selection; and Context Compression, which performs Relational Logic Distillation (RLD) to transform raw text into explicit entity-relation triplets. These relational triplets serve as a concise and structured prompt for final answer prediction. Extensive experiments on the OK-VQA and A-OKVQA benchmarks demonstrate that SCoRe consistently outperforms state-of-the-art methods.
KBMR is proposed, the first MLLM-based embedding retriever tailored for KB-VQA, and an MLLM-based semantic discriminator that generates continuous entity-consistency weights is introduced to tackle the challenge of noisy supervision in Wikipedia-scale retrieval.
This work proposes an inference-time evidence augmentation framework for frozen-LLM-based KB-VQA that focuses on how question-relevant multimodal evidence can be systematically constructed, refined, and organized before LLM inference.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
