GrayUI: A Visual Context Attack Framework Against VLM-Based Mobile Agents

Abstract

Recent advances in VLM-based GUI agents accelerate mobile automation but also introduce security risks, as agents rely on broad system privileges and long-horizon contextual continuity. Existing attacks, such as overlay manipulation and inference-gap exploitation, target individual steps but overlook a more fundamental vulnerability: the integrity of contextual observations across extended interaction trajectories. In this work, we argue that disrupting contextual coherence is a highly effective and scalable strategy for compromising mobile agents. The key challenge is crafting plausible perturbations that continuously mislead the agent without breaking visual realism. To this end, we propose GrayUI, a systematic visual context attack framework that exploits three dimensions of contextual dependency: 1) spatial context, by embedding adversarial prompts into wallpapers and persistently occupying overlay windows to pollute the visual field; 2) temporal context, by interrupting task trajectories through splash-screen advertisements; and 3) functional context, by confusing button semantics and introducing in-app pop-ups that distort intent-to-action mapping. In this way, by exploiting system-level (e.g., wallpapers) and application-level (e.g., advertisements) visual channels, GrayUI enables comprehensive context deception from the OS background layer to the app interaction layer. Finally, we evaluate GrayUI on two public benchmarks, showing that a joint attack combining all strategies achieves an average attack success rate of 86%.