It is shown that aligned vision-language models also condition refusal on a property of a request's form: whether an image is attached, holding everything the request asks fixed, which shifts benign refusal by tens of points.
Haoyu Zhang, Yi Feng, Han-Wen Liu et al.· 0 citations
This paper shows that Vision-Language Models (VLMs) can internally distinguish when abstention is required, but fail to express it anyway, and introduces Penalized Epistemic Calibration Score (PECS), a new robust metric that requires models to both answer correctly when the outcome is knowable, and abstain when the out...
Fnu Pramono, J. Cai, Sourabh Kulkarni· 5 citations· ⚡1
PARITY (Paired Assessment with Reused Identity), a synthetic, demographically balanced set of identity-controlled neutral/expression portrait pairs with neutral-neutral controls, is introduced, finding a clear split between refusal-first models and speculative models; in the latter, certain expressions disproportionate...
Karan Dua, Amit Agarwal, Hitesh Laxmichand Patel et al.· 0 citations
This work studies the problem in web-agent guardrails, where a VLM judges whether on-screen text conflicts with a user instruction, and proposes two training-free interventions that improve grounding without changing verdicts.
Language models hold latent quantities in a form they can report on, and more of a quantity is present in that form when the task requires reusing it flexibly when the task requires reusing it flexibly.
Vision-language models (VLMs) are expected to respond helpfully to appropriate requests while withholding compliance with requests that are incorrect, unsafe, infeasible, or unanswerable. However, existing benchmarks predominantly evaluate non-compliance at the level of the query as a whole, assuming that each request...
Minji Kim, Jihyoung Jang, Hyounghun Kim· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.