Execution-Aware Stage-Wise Evaluation Framework for Indirect Prompt Injection in LLM-Based Email Agents

Abstract

Indirect prompt injection (IPI) attacks have emerged as a critical security threat to large language model (LLM)-based email agents. However, existing evaluation benchmarks largely rely on simulated environments that cannot verify whether an attack will actually succeed in practice, often determining attack success solely based on tool invocation, which leads to frequent misclassifications of attack outcomes. In this paper, we propose execution-aware stage-wise evaluation (EASE) as an automated framework that quantitatively assesses the vulnerability of email agents to IPI attacks in a realistic environment integrated with the Gmail application programming interface (API) and multiple LLM APIs. EASE automates the entire pipeline of delivering attack emails, triggering the LLM agent, and collecting tool invocation logs, and judges attack success by whether the delivered email actually fulfills the attacker’s intent, using tool invocation only as a proxy baseline. Additionally, EASE’s modular architecture allows new LLMs, defense prompts, and attack datasets to be plugged in or replaced independently, and a Streamlit-based web interface makes the framework accessible to non-developers. To validate EASE’s effectiveness, we conducted 15,336 experiments using 639 attack samples across six LLMs under four defense conditions. The results demonstrate that executing attacks in the live service reveals what proxy-based judgment misses: 13.9% of proxy-judged successes are actually failures when the execution outcome is verified. We also observe that the same defense prompt can paradoxically increase the attack success rate for certain LLMs, underscoring the need for execution-grounded evaluation tools such as EASE. EASE is released as an open-source package on GitHub so that email agent developers and security researchers can adopt it immediately.