Backdoor Attacks on CLIP via Prompt Learning.

Abstract

Recent studies have identified vulnerabilities in contrastive vision-language pre-training (CLIP) models to backdoor attacks. Attackers manipulate models to exhibit targeted behaviors upon encountering specific triggers while maintaining normal performance on clean inputs. Existing backdoor attacks on CLIP primarily focus on the pre-training stage or require extensive data for fine-tuning, which limits their practicality in few-shot transfer learning scenarios. In this work, we introduce BadCLIP, a novel backdoor attack that leverages prompt learning to effectively inject malicious behaviors into CLIP models even in data-scarce scenarios. Our BadCLIP operates under two different scenarios. (1) BadCLIP-W assumes white-box access to models and consists of a learnable trigger and a trigger-aware context generator. To optimize performance, we introduce a trigger warm-up strategy during training. (2) BadCLIP-B functions under black-box constraints to models. To enhance attack performance, we first generate a trigger pattern by a black-box optimization process due to the absence of gradient information. Then, we adopt the trigger pattern to poison the training datasets to inject backdoors via prompt learning. Comprehensive experiments demonstrate that BadCLIP-W and BadCLIP-B achieve high attack success rates while maintaining comparable accuracy on clean images.