HDG-CLIP: Hierarchical Dual-Granularity Vision-Semantic Alignment for Open-Vocabulary Multi-Label Image Classification

Abstract

Open-Vocabulary Multi-Label Image Classification (OV-MLIC) is an emerging task in computer vision aimed at recognizing unseen categories in real-world scenarios, leveraging Vision and Language Pre-training (VLP) models like CLIP. However, existing methods overlook the impact of category coupling and scale variation on cross-category knowledge transfer, thereby restricting performance on unseen categories. To address this issue, we propose a novel OV-MLIC method called Hierarchical Dual-Granularity Alignment-CLIP (HDG-CLIP), which emphasizes the complementary characteristics of different modalities and introduces a sample-category matching mechanism. Specifically, to address the category coupling issue, we construct semantic category prototypes to enhance cross-category knowledge transfer. Through the interaction between visual embeddings and category prototypes, we decouple category-specific information from mixed visual features and leverage the visual context of samples to learn category-level visual features. For mitigating the scale variation issue, we build a sample-category dual-granularity matching mechanism based on the difference in capture capability of different modalities across scales, thereby improving the object localization accuracy from a multi-dimensional perspective. Extensive experimental results show that HDG-CLIP exhibits state-of-art performance over existing methods on both the NUS-WIDE and the Open-Images datasets. Our code is available at https://github.com/wakihy/HDG-CLIP