RoFLIP: Robust and Fine-Grained Alignment for Vision-Language Compositional Reasoning
TL;DR
The Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP) is proposed, enhancing both the robustness and granularity of vision-language alignment and underscore RoFLIP’s compositional reasoning and generalization abilities.
Abstract
Vision-language models pretrained on large-scale datasets have become the foundation of cross-modal understanding. However, they often overemphasize salient objects and underperform in compositional reasoning that additionally requires fine-grained perception of attributes and inter-object relations. Although compositional data-driven approaches have shown promise in addressing this challenge, low-quality captions undermine robust alignment and consequently hinder fine-grained perception. Specifically, two fundamental issues arise: (1) low-density positive captions restrict the global representation capability of image encoders, causing local alignment. (2) false-negative captions introduce noise into training, resulting in fragile alignment. In response, we propose the Robust and Fine-grained training framework for CLIP-based vision-language models (RoFLIP), enhancing both the robustness and granularity of vision-language alignment. To address the first challenge, RoFLIP employs the Densified Ranking Loss (DRL), which promotes global visual representations by contrasting dense and sparse captions. For the second challenge, RoFLIP incorporates Denoised Cross-modal and Text-modal Contrastive Losses (DCCL &DTCL), reducing the impact of noise while effectively leveraging negative captions. In addition, we introduce RoFLIP’s Auxiliary dataset (RoFA), comprising 225K high-quality captions that are fluent and image-related to support training. Quantitative evaluations demonstrate that RoFLIP-CLIP-B/32 achieves an average gain of 7.41%\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$7.41\%$$\end{document} across 5 compositional reasoning benchmarks while maintaining considerable performance on 23 downstream benchmarks. Furthermore, replacing the corresponding modules in LLaVA-v1.5 and Stable Diffusion v1.5 with RoFLIP-CLIP-L/14 also brings performance gains on VQA and image generation tasks, respectively. These results underscore RoFLIP’s compositional reasoning and generalization abilities.