We report a counter-intuitive interaction between image inputs and existing black-box defenses on Vision--Language Models (VLMs): pairing an encoded jailbreak prompt with an unrelated decoy image can sharply lower attack success rate (ASR). The operative change is in the defense pipeline, not in the image. Across five frontier VLMs, two encoded-attack families, and three black-box defenses, a caption-mediated defense (ECSO) that leaves ASR essentially unchanged on text-only encoded input drops it by up to $73$pp once a content-free decoy is attached; every non-saturated contrast is significant under exact McNemar tests. We advance two hypotheses for this pattern, supported by indirect evidence rather than pipeline introspection, since a black-box threat model precludes inspecting vendor internals: caption-mediated defenses branch on image presence, and intrinsic image-side safety engages on image-resident content. Three controls constrain the explanation. Blank-canvas and natural-photograph decoys reproduce the effect on every model, implicating image presence rather than content; the effect replicates on three open-weight VLMs served with no moderation layer, so it is not a vendor-filtering artifact; and a non-symbolic, meaning-based encoder reproduces it, so it is not specific to symbolic obfuscation. Attaching a decoy unconditionally is not deployable --- it raises benign refusal to $20$--$79\%$, an inflation of $+10$ to $+67$pp --- but gating attachment on a lightweight encoded-input detector returns benign refusal to the text baseline while preserving the safety gain wherever the detector fires, making detector recall the binding constraint. Under adaptive attacks that target the caption-mediated re-check, the effect degrades but holds. We frame this as an observation about pipeline interaction, not as a robust defense.
Haoyu Zhang, Xiang-Chen Guan, Shi-Bo Zheng et al.· 0 citations
Safety classifiers ("guards") are the dominant black-box defense for vision-language models, yet a guard judges an input's surface form, not its meaning: a harmful request re-encoded as set theory, formal logic, a classical language, code, or text rendered inside an image slips past a guard that would block it in plain language - the decode gap. The standard fix is a preprocessor that recovers image content and decodes the encoding before the guard. We build one and evaluate it against an ensemble of eleven published encoding attacks, counting a behavior as broken if any attack succeeds. That metric separates two mechanisms such defenses conflate. Restoring a view the guard never had improves it on both axes at once: it blocks far more attacks, and, measured on a category-balanced benign set, it blocks fewer benign requests, because restating a request normalizes the borderline phrasing a classifier over-flags. It still does not make the system safer: against an attacker free to choose among eleven encodings, closing one channel relocates the success rather than removing it, and no ensemble contrast survives multiple-comparison correction. What does lower ensemble attack success is re-screening the recovered pre-decode surface, and that step is where the entire benign cost falls. The safety-utility trade-off is therefore not a property of recovery; it is localized to one step. Across the full guard x target x condition factorial, no configuration reaches an ensemble attack-success rate at or below 40% while holding benign over-refusal under 70%. The per-attack averages usually reported understate the attacker roughly fourfold, which is why this frontier is easy to miss. Composing across defense families is the one lever that moved the safety axis, beating every configuration we measured, and still landing far outside any deployable refusal budget.
Haoyu Zhang, Zhuo-Xiang Wang, Shi-Bo Zheng et al.· 0 citations
A coding agent that installs packages and untangles version conflicts is implicitly reasoning about semantic-versioning constraints and dependency resolution. Whether current language models can actually do this has not been measured, and that is the gap we address. DepResolve-Bench is a programmatically generated benchmark scored by a deterministic oracle that needs no human labels. We trust the oracle: it agrees with the production node-semver on 2250 sampled cases and, for resolution, with exhaustive search on 3000 instances. The benchmark has two tasks: deciding whether a version satisfies a range over the full npm grammar (Task A), and resolving a multi-package dependency graph or proving it unsatisfiable (Task B), with instance size as a difficulty knob. Five frontier and lightweight API models agree. They have nearly mastered the atomic rules, with Task A macro accuracy spanning 0.92 to 1.00, yet they cannot reliably put those rules together. On resolution the best model reaches 0.91 and the weakest 0.29, and every model degrades as the graphs grow, the strongest falling to 0.82 on the largest instances. A missed satisfiable instance almost always breaks a transitive constraint; on an unsatisfiable one, weaker models invent a resolution anyway (GPT-4o-mini for 44 of 80 such cases, the strongest model for one). How much we let a model reason matters more than which model it is: capped at 200 output tokens every model sits near 0.1, and accuracy climbs as the budget grows. We release the generators, oracle, and item banks; the results argue for keeping a real resolver in the loop rather than handing dependency management to a language model.
Zhuo-Xi Wang, Haoyu Zhang, Jing-Wen Hou et al.· 2026 8th International Confe...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
