Shubhankar Mahanta

The Margin Knows the Background: Uneven Abstention in Frozen Vision-Language Models

Selective prediction allows a model to abstain on uncertain inputs, but a single confidence threshold can disproportionately reject examples from particular groups. We study how confidence scores produce this imbalance in frozen vision-language models (OpenAI CLIP, LAION CLIP, SigLIP). We express a family of scores, in...