Entropy-Adaptive Patch Weighting for Improving Gradient-Based Adversarial Attack on Vision Transformer

Abstract

Adversarial attacks on Vision Transformers (ViTs) commonly rely on selecting a subset of important patches to guide perturbation allocation, with hard top-k strategies being a widely used approach. However, such methods convert continuous attention distributions into binary decisions, potentially discarding useful information when patch importance scores are similar. To address this limitation, we propose a soft attention-based patch selection framework that assigns continuous importance weights to all patches, enabling more flexible and fine-grained perturbation allocation. Building on this idea, we further introduce an entropy-adaptive weighting mechanism that dynamically adjusts the sharpness of the weighting distribution according to the entropy of attention scores. By adapting to both concentrated and dispersed attention patterns, the proposed approach removes the need for manual tuning of the sharpness parameter. We evaluate our method on CIFAR-10 using a ViT-B/16 model under a white-box Projected Gradient Descent (PGD) attack setting. Experimental results show that the proposed entropy-adaptive strategy achieves a high Attack Success Rate (ASR) of 97.3%, substantially outperforming the hard top-25% baseline (89.7%) while maintaining comparable perceptual quality (PSNR ≈ 34 dB). Furthermore, the adaptive mechanism provides strong attack performance without requiring dataset-specific tuning of weighting parameters. These findings suggest that entropy-aware weighting offers a flexible and robust approach for patch-level perturbation allocation in adversarial attacks on Vision Transformers.