ClassifierAlign: Unsupervised multi-label image classification via training-free vision-language classifier alignment.
Abstract
Existing vision-language models (VLMs) exhibit strong zero-shot generalization, yet remain fundamentally limited in Unsupervised Multi-label Image Classification (UMIC), primarily due to their inadequate multi-object perception and the intrinsic modality gap between vision and language. In particular, global representations tend to entangle multiple concepts, hindering accurate recognition of co-existing objects. To address these challenges, we revisit UMIC from a decomposition-then-classification perspective. Instead of predicting multiple concepts holistically, we first decompose an unlabeled multi-label image into multiple single-concept visual instances, thereby explicitly improving multi-object perception. We then perform classification based on aggregated predictions. Building upon this paradigm, we propose ClassifierAlign, a fully training-free framework that jointly leverages visual foundation models (VFMs) and VLMs. Specifically, we derive a lightweight visual classifier directly in the visual feature space, and semantically align it with CLIP's textual classifier, effectively bridging the vision-language modality gap without updating parameters of the pretrained foundation models. This design further enables seamless integration of powerful self-supervised VFMs (e.g., DINO), yielding robust and transferable representations without requiring spatial alignment. Extensive experiments demonstrate that ClassifierAlign achieves competitive performance with fine-tuning-based methods, while significantly reducing computational overhead and eliminating the need for costly annotated data. The source code for this work can be found at https://github.com/dengxili/ClassifierAlign.