Ratan J. Sebastian

Using Hierarchical Controlled Vocabularies to Understand CLIP Retrieval Failures in Historical Photo Collections

It is found that visual coherence and text-image alignment are nearly uncorrelated across terms and jointly separate distinct failure modes, and fine-tuning improves retrieval overall, but its gains favour shallower terms in the hierarchy, where text-image alignment improves most, beyond what concept frequency explains.