Vision Language Models for Cross-Modal Retrieval

Abstract

Contrastive Language–Image Pre-training (CLIP) gives strong zero-shot baselines for cross-modal retrieval, but full fine-tuning is costly and InfoNCE treats every non-matching pair as an equally wrong negative. We study parameter-efficient fine-tuning of CLIP ViT-B/32 on MS-COCO 2017 with three components added incrementally: (i) Low-Rank Adaptation (LoRA) of the query/value projections (0.32% trainable parameters); (ii) FAISS-mined hard negatives with a skip-top-3 rule and a curriculum; and (iii) an entailment-aware loss using DeBERTa-v3 natural-language-inference (NLI) scores as soft labels. The best model reaches image-to-text Recall@1/5/10 of 61.82/84.98/91.20% on COCO 5k, +11.86 R@1 over zero-shot, with no inference overhead. Two components, however, do not behave as first assumed, and we report them as diagnosed negative results. The logs show that the whole gain was reached before mined negatives were activated; the curriculum moved R@1 by +0.1. A false-negative audit of the mined pool (10.1 M anchor–negative pairs) explains why: 17% of the negatives used in training carry a near-duplicate of the anchor caption and 58% of a manually inspected sample (75% counting borderline cases) are plausible matches, so skip-top-3 is insufficient for COCO’s generic captions. Second, NLI scores exist only for captions of the same image, so under random batching the soft-label matrix is uniform in 98.6% of steps and the KL term reduces to a confidence penalty; this explains the degradation (55.38% at w = 0.5, 41.18% at w = 0.1) without any conflict between NLI and contrastive alignment. Every number carries its evaluation protocol.