Examining Racial and Gender Bias in CLIP via Explainability-Based Visual Attention Analysis

Abstract

Vision-language models (VLMs) such as CLIP are increasingly integrated into automated decision-making systems where unexamined social biases can directly harm racial and gender minorities through discriminatory outcomes and unjust content censorship. Despite strong multimodal performance, CLIP's susceptibility to demographic bias remains underexplored from a visual attribution perspective. Consequently, the mechanisms by which these biases emerge in visual processing remain opaque, both to researchers and to governance bodies concerned with the equitable and transparent deployment of AI. We address this diagnostic gap by applying four attribution methods to the PAIRS dataset, a controlled benchmark of parallel occupation images balanced across race and gender. We then seek to determine whether demographic stereotyping is detectable and measurable at the level of visual attention, rather than only through output statistics. Preliminary results are mixed: subject-level differences in face-focused saliency across race conditions are not statistically separable at the aggregate level, but under race-explicit prompting we observe the clearest race-conditioned saliency differences. These results suggest that attribution-based diagnostics can expose hidden visual biases, invisible to standard fairness metrics, and help explain how demographic bias emerges in multimodal AI systems. Furthermore, we comparatively analyze the behavior of the attribution methods and discuss their outputs with respect to the broader implications of visual bias diagnostics for fairness and transparency in deployed multimodal AI systems.