CaptchaBench: A Modality-Stratified Benchmark Dataset for Evaluating Adversarial Perturbation Against VLM-based CAPTCHA Recognition

Abstract

Vision-Language Models (VLMs) now achieve over 99% zero-shot accuracy on Chinese-character CAPTCHAs, rendering traditional hardening strategies (complex backgrounds, stylized fonts, character distortion) ineffective. Adversarial perturbation offers an emerging defense by exploiting model vulnerabilities at the decision-boundary level, yet existing methods originate from disparate communities and have never been compared under identical conditions; moreover, the role of perturbation input modality has not been systematically characterized across heterogeneous VLM architectures. To bridge these gaps, we introduce CaptchaBench, a large-scale benchmark comprising 840,000 base Chinese-character CAPTCHA images (3,500 characters sampled from GB2312 Level-1 × 120 backgrounds × 2 generative pipelines), with a stratified evaluation subset of 16,000 adversarially perturbed images. Eight perturbation methods are organized into three input-modality groups (Image-only: ASPL, Glaze, AMP, XTransfer, AnyAttack; Text-only: Nightshade; Image+Text: MMCoA, CoA) and assessed under a unified three-question framework: Q1 Direct-Recognition Accuracy (DRA) and Q1 Failure, Text Visibility Rate (TVR), and Q3 Attack Success Rate (ASR), capturing complementary black-box response behaviors. Baseline evaluation across five commercial VLMs (Qwen-VL, Kimi 2.5, GPT-5.2, Gemini-3.0, GLM-4V) yields three key observations: (1) perturbation input modality is associated with distinct black-box response patterns: image-only methods achieve high visual confusion but variable VLM-reported visibility, while methods differ in cross-VLM variation; (2) Gemini-3.0 exhibits a pronounced, model-specific TVR response pattern (62.4--97.3% vs. ?20.8% for all other VLMs; ≥3.0× vs. the highest non-Gemini TVR across all eight methods), that persists across prompt variants; and (3) stroke-bin Q3 ASR varies across methods, with the 16--20-stroke bin high in this evaluation; the sparsely sampled 21+ bin is reported descriptively. This motivates character structure as a CAPTCHA design dimension. A 29-participant four-choice study confirms high human readability of perturbed CAPTCHAs (98.9% accuracy). The dataset and evaluation code are released under CC BY 4.0. Extended tables, figures, and visual case studies are available as online supplementary material on the CaptchaBench dataset website at https://github.com/Linzy19/CaptchaBench.