Dual-stream Soft Prompt Attacks on VLMs: Coordinated Visual and Textual Prompt Injection for Stealthy Backdoors
TL;DR
This work is the first to expose how vulnerable vision–language models (VLMs) are to soft-prompt backdoors and introduces the Dual-stream Soft Prompt (DSP) Attack.
Abstract
Large vision–language models (VLMs) have become increasingly powerful, but their security risks have risen just as sharply—especially backdoor attacks, in which a model behaves normally on clean inputs yet produces malicious outputs when a hidden trigger is present. To our knowledge, we are the first to expose how vulnerable vision–language models (VLMs) are to soft-prompt backdoors and introduce the Dual-stream Soft Prompt (DSP) Attack. DSP inserts tiny soft prompt channels into both the visual and textual branches, enabling output manipulation while keeping the entire backbone frozen. We further design two dual-stream prompt variants: Orthogonal Prompts (OP), where visual and textual prompts are independent, and Mutual-Guided Prompts (MGP), where the two prompts are directionally aligned. Experiments on several mainstream VLM architectures and public datasets show that DSP requires only a small poisoning budget and minimal additional parameters, yet markedly increases both single-trigger and dual-trigger attack success rates. These findings expose an overlooked threat surface and establish a stronger baseline for future research in defending against backdoor attacks on VLMs.