An automated framework is developed that searches application-specific perturbation strategies, optimizes their content using representative tasks, and realizes the perturbations in deployable applications, showing that inconspicuous application content can redirect delegated actions even without runtime adaptation.
Abstract
Mobile agents may act on interface content that users cannot see or easily notice, making their actions harder to oversee. We characterize this gap as human-agent UI desynchronization and investigate how attackers can exploit it through perturbations embedded in applications before deployment. Beyond accessibility metadata, we identify two mechanisms that expose misleading cues to screenshot-based agents: content rendered in display-occluded regions under compatible device configurations, and low-contrast text that remains encoded in screenshots but is difficult for users to notice. We develop an automated framework that searches application-specific perturbation strategies, optimizes their content using representative tasks, and realizes the perturbations in deployable applications. The perturbation content remains fixed at runtime and requires neither access to future user instructions nor detection of agent use. Using a benchmark of 546 tasks across 13 applications, we evaluate five mobile-agent frameworks and three backbone models. The attacks achieve average misleading rates of 77.4% in static observation-space evaluation and 67.2% in dynamic application-level evaluation, where success denotes selection of an attacker-designated decoy control. We invited 195 participants to assess the perceptibility of our visual perturbations and found that none spontaneously noticed the low-contrast text. These findings expose a security gap between human-visible interfaces and agent-observable representations, showing that inconspicuous application content can redirect delegated actions even without runtime adaptation.
Mobile GUI agents read the screen of a phone and operate apps on a user's behalf to complete multi-step tasks such as online shopping or messaging. Because the agent acts on whatever the screen shows, an attacker can embed malicious instructions into ordinary UI content and redirect its actions. We refer to this threat...
Ying-Zi Jin, Ji'an Wang, Hao-Yang Chai et al.· 0 citations
Web services increasingly expose backend interfaces that let users delegate tasks to AI agents, which invoke service operations directly without navigating websites. This shift changes how existing Web vulnerabilities can be exploited. We investigate these changes through VELA, an executable benchmark covering 44 vulne...
Ruo-Zhao Yang, Jiongchi Yu, Ming Hu et al.· 0 citations
Modern OS kernels expose hundreds of syscalls to applications, yet individual programs typically require only a small subset. This over-privileging violates the principle of least privilege and expands the kernel attack surface. Existing methods for limiting system calls face several challenges: dynamic tracing suffers...
Dong-Yang Zhan, Xing-Chen Yan, Zhao Yu et al.· Proceedings of the 41st IEEE...· 0 citations
LLM-powered browser agents are transforming web automation by autonomously interacting with websites and executing complex workflows. However, the effectiveness of existing anti-bot defenses against these systems remains unclear. We present a systematic empirical evaluation of modern agentic browsers against widely dep...
Moiz Ali, Nafis Karim, Rownak Islam et al.· 0 citations
ProtoArena is introduced, a black-box, multi-component, dynamic benchmark with 80 cross-component vulnerability-exploitation environments spanning 11 protocol and application domains and 7 attack categories and success with limited black-box information remains low and varies across domains and attack categories.
Fa-Sheng Miao, Zhao-Yu Hu, Xiang Li et al.· 0 citations
This work introduces AgentDecept, an executable benchmark for evaluating behavioral deception under pressure that enables controlled analysis of when multi-turn agents'reports diverge from execution in realistic application interactions.
Ye-Han Zhang, Qian-Qi Yan, Hang Yin et al.· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.