CertifiedGPT: Evaluating Adversarial Robustness of Vision and Language Models Against Targeted Black-Box Attacks

Abstract

Multimodal Vision and Language Models (VLMs) have attracted significant academic interest due to their strong performance in tasks that require simultaneous inference across image and text modalities. The impact of VLMs is evident both in general society, through powerful chatbots such as ChatGPT and Google Gemini, and in specific industry sectors, including industrial processes, biomedical engineering, systems engineering, and medicine. In contrast to proprietary models, open-source VLMs such as MiniGPT-4 have emerged, offering compact alternatives that demonstrate strong performance. However, such VLMs show certain vulnerabilities to noisy input data (e.g., images), which can compromise the reliability of the output and can be explored by adversarial attacks, even under black-box access. More sophisticated attacks can also force a VLM to generate specific output using targeted attack configurations. In this work, we explored the concept of robustness certification in order to make a VLM robust against targeted black-box attacks through the use of a randomized smoothing technique. First, we fine-tuned the MiniGPT-4 model on a small subset of VQAv2 and applied Gaussian noise to the input images. We also adapted a smoothed method that encapsulates the original decoder and generates the most probable answer, even with the noise in the input images. Finally, we evaluated the model against targeted black-box attacks. Our certified version of MiniGPT-4, when evaluated on a small VQAv2 subset, produced statistically significant results, demonstrating that randomized smoothing is a feasible approach to certifying the robustness of VLMs, especially in scenarios where access to high-performance GPU is limited.