Debiased visual question answering via spatial-frequency invariant semantic learning
TL;DR
A novel spatial-frequency invariant semantic learning model designed to overcome limitations in visual question answering models by incorporating frequency-domain features as an additional modality and employing invariant feature learning techniques, effectively reduces bias without relying on external datasets.
Abstract
Visual question answering models often encounter challenges related to data biases and exhibit limited performance in specialized domains such as cultural heritage, where recognizing fine-grained textures is crucial. In this study, we propose a novel spatial-frequency invariant semantic learning model designed to overcome these limitations. By incorporating frequency-domain features as an additional modality and employing invariant feature learning techniques, our model effectively reduces bias without relying on external datasets. The proposed model extracts invariant representations across textual, spatial, and frequency domains, thereby filtering out spurious correlations. Comprehensive experiments on benchmark datasets, including VQA-CP v2 and GQA-OOD, demonstrate that our model achieves state-of-the-art results. Furthermore, our model exhibits enhanced robustness when applied to cultural heritage datasets, proficiently handling complex visual textures and multimodal reasoning tasks. This model enhances the capabilities of visual question answering systems in identifying artistic materials and techniques, providing a robust solution tailored to domain-specific applications.