PKSF: A Task-Aware Prior Knowledge Selection and Fusion Framework for TextVQA
TL;DR
This work proposes a TextVQA framework that integrates external prior knowledge to support multimodal reasoning and demonstrates that this approach effectively leverages external knowledge to improve performance on knowledge-intensive questions.
Abstract
Text-based visual question answering (TextVQA) requires reasoning over images containing rich textual content, often involving knowledge beyond what is directly observable. Existing methods fuse visual objects and OCR tokens but struggle when questions require external knowledge. Moreover, naively incorporating retrieved knowledge often introduces irrelevant or misleading information, which may hinder reasoning rather than support it. To address these challenges, we propose a TextVQA framework that integrates external prior knowledge to support multimodal reasoning. Given an image and question, a task-aware knowledge retrieval module selects relevant candidates, which are then filtered and verified by a knowledge verification module leveraging large language models. The verified knowledge and question are compressed into compact embeddings via a perceiver-based semantic resampler and jointly processed with visual and OCR features in a multimodal reasoning module. Experiments on the TextVQA and ST-VQA datasets demonstrate that our approach effectively leverages external knowledge to improve performance on knowledge-intensive questions.