Multi-modal Boundary Testing of Vision–Language Models

Abstract

Vision-language models (VLMs) are increasingly deployed in settings where failures can have costly consequences, making it important to identify inputs for which small changes flip their behaviour. Boundary testing searches for inputs near such behavioural boundaries and is well established for discriminative AI systems, but it requires a signal of boundary proximity: a classifier's confidence provides such a signal, whereas a VLM's generative interface does not. We construct a signal by force-decoding a fixed set of candidate answers. The contrast between their length-normalised log-probabilities serves as a boundary-proximity score, making the boundary searchable for any behaviour formulated as a choice between candidate answers. Using this signal, we develop a framework that manipulates the image and question text separately. We evaluate it on two VLM families in zero-shot classification, where the difficulty of a boundary proves specific to the model: it follows each model's own preference for one of the two answer words. Image manipulations produce boundary crossings that survive human assessment, whereas text manipulations reach boundaries more readily but more often erode semantic fidelity.