One Geometry, Different Outcomes: Readout-Dependent Effects of the Modality Gap in Vision-Language Models
Contrastive vision-language models learn shared embedding spaces by aligning matched image-text pairs, yet their representations remain separated by a modality gap. Prior work reports divergent effects of modifying this gap: reducing it can improve zero-shot classification and cross-modal alignment, whereas removing ga...