INSURE: Inference-Time Defense Framework for Securing Agents against Hijacking

Abstract

Agents are increasingly susceptible to hijack attacks, where malicious instructions embedded in cross-modal inputs (e.g., adversarial images or retrieved web content) are used to deceive the agent's execution. Existing defenses typically rely on strict filtering or refusal mechanisms, which suffer from a critical limitation: over-refusal, prematurely terminating valid agentic workflows. We propose INSURE, a probing-to-mitigation framework that neutralizes cross-modal attacks while preserving task continuity. Our key insight is that multimodal IPI attacks leave distinct over-focusing signatures in the vision-language latent space. We introduce a Latent Space Trace Prober to detect inter-modal attacks based on high-intensity activation scores. Subsequently, a Mitigating Rectifier performs surgical attention steering that selectively manipulates adversarial visual-textual dependencies while amplifying task-relevant elements to restore the large model's functional trajectory. Extensive evaluations on multiple multimodal backbones show that our method achieves a competitive 0.3% ASR (average), matching commercial-grade detectors, while yielding over 50% task utility gain. Furthermore, INSURE demonstrates robust Out-of-Distribution (OOD) generalization and seamlessly secures diverse multimodal interactions, establishing a superior balance between security and efficiency.