Recent studies have identified vulnerabilities in contrastive vision-language pre-training (CLIP) models to backdoor attacks. Attackers manipulate models to exhibit targeted behaviors upon encountering specific triggers while maintaining normal performance on clean inputs. Existing backdoor attacks on CLIP primarily focus on the pre-training stage or require extensive data for fine-tuning, which limits their practicality in few-shot transfer learning scenarios. In this work, we introduce BadCLIP, a novel backdoor attack that leverages prompt learning to effectively inject malicious behaviors into CLIP models even in data-scarce scenarios. Our BadCLIP operates under two different scenarios. (1) BadCLIP-W assumes white-box access to models and consists of a learnable trigger and a trigger-aware context generator. To optimize performance, we introduce a trigger warm-up strategy during training. (2) BadCLIP-B functions under black-box constraints to models. To enhance attack performance, we first generate a trigger pattern by a black-box optimization process due to the absence of gradient information. Then, we adopt the trigger pattern to poison the training datasets to inject backdoors via prompt learning. Comprehensive experiments demonstrate that BadCLIP-W and BadCLIP-B achieve high attack success rates while maintaining comparable accuracy on clean images.
Kuofeng Gao, Jiawang Bai, Shaobo Min et al.· IEEE Transactions on Pattern...· 0 citations
Reasoning in Multimodal Large Language Models (MLLMs) requires both fine-grained visual perception and rigorous logical deduction. Explicit text-based Chain-of-Thought (CoT) is computationally expensive and prone to visual hallucinations, while existing latent reasoning methods typically require costly training. Furthermore, directly adapting training-free LLM reasoning mechanisms to the multimodal setting yields unstable performance. We identify that this failure stems from their reliance on token-level entropy, which fundamentally conflates perceptual ambiguity (e.g., unclear visual details) with logical uncertainty (e.g., complex reasoning steps). To overcome this bottleneck, we present a novel training-free inference strategy for MLLMs that explicitly decouples perception and reasoning. We propose a novel metric, the vision-to-text attention ratio, to dynamically gauge the model's cognitive focus. Guided by this metric, our proposed framework, Attention-Guided Switching (AGS), adaptively triggers latent reasoning for perceptual tokens to preserve high-fidelity visual information in the continuous space, while enforcing explicit text generation for logical tokens to maintain structural anchoring. Extensive experiments demonstrate that our method achieves state-of-the-art performance, significantly improving both accuracy and inference efficiency by reducing autoregressive steps and latency. Code is released at https://github.com/swordAndSnow/MM26-AGS.
Haoqiang Kang, Liupeng Li, Kuofeng Gao et al.· 0 citations