A systematic review and structured descriptive synthesis of research on defenses against prompt-based attacks in language model and agent systems reveals trade-offs between security effectiveness, performance, and system complexity as well as major gaps in benchmarks, indirect attack coverage, and multi-agent evaluation.
Abstract
Current language model deployments face growing security challenges from prompt-based attacks, including jailbreaks, direct and indirect prompt injection, and instruction hijacking, which often evade traditional rule-based safeguards. As these models are increasingly integrated into agent-based systems, retrieval pipelines, and tool-driven workflows, such attacks exploit their natural language interfaces to bypass safety constraints and manipulate system behavior, in some cases leading to data leakage or unauthorized actions. Recent research has proposed a wide range of defense mechanisms ranging from prompt-level filtering and model-level detection to pipeline wrappers and multi-agent protection frameworks. Many of these methods report strong results in controlled experiments, yet their effectiveness depends heavily on the assumed threat model, the datasets used, and the evaluation protocol. This paper presents a systematic review and structured descriptive synthesis of research on defenses against prompt-based attacks in language model and agent systems. By integrating and comparing findings across multiple studies, we identify major attack categories, commonly adopted defense strategies, deployment stages, and evaluation trends, while also highlighting limitations related to generalization, robustness, and real-world applicability. The analysis reveals trade-offs between security effectiveness, performance, and system complexity as well as major gaps in benchmarks, indirect attack coverage, and multi-agent evaluation. The systematic review concludes by outlining future research priorities, including pipeline-aware defense design, adaptive and layered protection mechanisms, and more realistic evaluation practices to support the development of robust and deployable prompt security solutions.
Large Language Model (LLM)-based agents automate complex tasks by integrating external tools such as web browsers, e-mail clients, file readers, and APIs, but this same integration exposes them to indirect prompt injection (IPI) attacks, in which malicious instructions hidden in tool content hijack the agent. A central but often overlooked question is how defending against such attacks affects the LLM and its own task performance and computational efficiency. In this study, we design a comprehensive testbed and a layered defense, Spotlight-Guard, that combines spotlighting-based input isolation, an LLM detection-and-quarantine pipeline, and instruction integrity based on a Hash-based Message Authentication Code (HMAC) into a single framework, and we evaluate it jointly along two axes: security and LLM performance. Experiments on locally hosted 7B-class open-weight models (Qwen-2.5-7B, Mistral-7B, and DeepSeek-Coder) use Attack Success Rate (ASR) for security and benign-task success rate together with confusion-matrix-based metrics (precision, recall, and F1) for task performance, all with bootstrap 95% confidence intervals. Across a stratified, fixed-seed benchmark of 250 adversarial and 250 benign cases per configuration, the full system reduces the ASR from 36.0% to 17.2% while preserving a 97.2% benign-task success rate and raising the detection F1 from 0.749 to 0.892, demonstrating that strong protection need not degrade the model’s task performance. A component ablation isolates each layer’s contribution, an adaptive-attack evaluation confirms a low ASR (6.7%) under attacks crafted to target the pipeline, and an analysis of computational cost (model invocations per request) quantifies the efficiency overhead, characterizing the security–performance trade-off of layered defenses on open-weight LLMs.
Doygun Demirol, Murat Aydoğan· Applied Sciences· 0 citations
Large Language Models (LLMs) with reasoning capabilities (e.g., DeepSeek-R1) gained substantial research and industry interest. However, their novel reasoning features may introduce vulnerabilities, especially to specific jailbreak attacks that exploit weaknesses in safety alignment. Despite growing awareness of the associated risks in related works, experimental evaluations of defensive mechanisms applied to reasoning models and the comparison with their non-reasoning versions are not yet available in the literature. The objective of this work is to evaluate the security of reasoning model DeepSeek-R1 against jailbreaks, compare it with the non-reasoning model DeepSeek-V3, and assess the effectiveness of two prompt-level defenses: Self-Reminder and Intention Analysis. We used a dataset of 75 jailbreaks with 10 malicious tasks, totaling 750 static attacks. The models were tested in three settings: 1) baseline (i.e., no defense), 2) using Self-Reminder, and 3) using Intention Analysis. Using automated classification with Llama-3.3-70B to measure the Attack Success Rate (ASR), we found that DeepSeek-R1 exhibited a baseline ASR of 70.27%, significantly higher than DeepSeek-V3 (53.47%). Results demonstrate that while Intention Analysis was more effective for DeepSeek-R1 (reducing ASR to 6.00%), Self-Reminder showed greater efficacy for DeepSeek-V3 (reducing ASR to 17.60%). As conclusion, the reasoning model DeepSeek-R1 was more susceptible to jailbreak attacks than the non-reasoning model DeepSeek-V3, and different prompt-level defenses were effective against static jailbreaks. As contributions, this work combines a focused literature review with a empirical evaluation to provide insights into the security of reasoning-based models and the effectiveness of two prompt-level defenses. Warning: this work contains inappropriate language in AI model outputs and jailbreaks.
V. T. Hayashi, Milton Pedro Pagliuso Neto, C. Miers et al.· IEEE Access· 0 citations
Large language models (LLMs) embedded in enterprise workflows cannot structurally distinguish legitimate instructions from adversarial ones in the same token stream, making prompt injection OWASP's top LLM risk for two consecutive editions a persistent threat across direct and indirect vectors. This paper presents PromptShield-RT, a layered, real-time, model-agnostic framework combining input normalization and provenance tagging, lexical-heuristic pattern matching, a statistical classifier, structural anomaly features, and calibrated risk fusion, with policy-driven mitigation (allow/sanitize/quarantine/block) and an explainable, adaptive-feedback mechanism for SOC workflows. We construct an original evaluation corpus, SynPI-Bench (n = 450, six categories), and a template-disjoint held-out generalization set (n = 31) with novel phrasings, obfuscation encodings, and adversarial hard-negative benign text. Using template-grouped 5-fold cross-validation, the fused pipeline achieves 92.4% accuracy (F1 = 0.930, AUC = 0.990), outperforming heuristic-only (57.0%) and naive-averaged (59.2%) baselines, while a lexical classifier reaches 85.9% with lower precision. We report a pronounced generalization gap on the held-out set (48.4% accuracy, 90% false-positive rate on hard negatives), quantifying a known limitation of surface-lexical defenses. The pipeline achieves sub-millisecond P95 latency (0.266 ms), within typical 50 ms enterprise SLAs. We situate PromptShield-RT relative to structural, architectural, and guardrail-product defenses, arguing for layered, defense-in-depth architectures, with reproducible code provided.
Fatimah Alhamzawi· Al-Noor Journal of Engineeri...· 0 citations
The deployment of Large Language Model (LLM)-generated SQL in Artificial Intelligence of Things (AIoT) systems introduces critical security risks, as prompt injection attacks can manipulate LLMs into producing unauthorized queries that expose sensitive data or execute destructive operations. Existing Natural Language to SQL (NL2SQL) research targets query accuracy, while current Model Context Protocol (MCP) servers offer only SQL-level protection without fine-grained, role-based access control. This paper proposes SecureMCP, a policy-enforced framework that integrates Role-Based Access Control (RBAC) with an MCP server to establish multi-layer defense for LLM-generated SQL execution. Grounded in an explicit threat model, the framework chains five defense modules in a sequential fail-closed pipeline addressing six prompt injection types spanning four adversary goals. We evaluate SecureMCP on the IoT-SQL dataset using Qwen3-8B, reporting filter performance—false positive rate (FPR) and false negative rate (FNR)—separately from LLM generation quality. On benign queries, the framework maintains a low false positive rate (0.3–2.2%) across four RBAC roles while keeping execution accuracy among allowed queries within 65.1–76.4%, matching the unprotected baseline of 63.8% and confirming that the defenses act as a transparent pre-execution filter. On 2400 adversarial queries, SecureMCP limits the effective false negative rate—computed over realized threats in which the injection payload was actually incorporated—to 3.98%, and an ablation confirms that RBAC and MCP-level defenses are complementary, as neither blocks the full range of injection vectors alone. The 72.5% injection incorporation rate confirms high LLM susceptibility, establishing the necessity of external policy enforcement.
Wonbae Kim, Hee-Kyong Yoo, Nammee Moon· Applied Sciences· 0 citations
Large Language Models (LLMs) have undergone a shift from stateless conversational interfaces to autonomous agents capable of multi-step planning, tool invocation, code execution, and maintaining persistent memory. When these agents operate with real-world privileges---calling APIs, modifying files, and querying databases---a compromised reasoning step can trigger unauthorized data access, irreversible state changes, or cascading failures, yet the security research community has not kept pace. To quantify the state of the field, we conducted a systematic literature review under PRISMA 2020 guidelines across six databases, screening 743 records and retaining 85 papers (2023--2025) on agentic LLM security. Attack research outpaces defense work by 3.9:1. Perception-layer vulnerabilities (prompt injection, jailbreaking, adversarial perturbations) dominate, accounting for 66\% of papers, while action-layer vulnerabilities (tool misuse, code injection, sandbox escape) appear in only 4.7\%, misaligned with real-world risk. Code execution security accounts for 3.5\%, and tool-augmented agents 12\%. We contribute a four-layer taxonomy mapping 13 vulnerability types across perception, brain, action, and interaction layers, and identify seven open problems centered on containment. Agentic LLM insecurity stems from architectural coupling, where weak isolation allows vulnerabilities to propagate across layers.
Md. Jafrin Hossain, Mohammad Arif Hossain, Nirwan Ansari· 0 citations
Prompt injection poses a significant security risk to Retrieval-Augmented Generation (RAG) systems, enabling adversaries to embed malicious instructions within retrieved documents and hijack model behavior to exfiltrate sensitive information or execute unauthorized actions. This work presents a modular dynamic evaluation environment for systematically testing and comparing defense mechanisms against prompt injection attacks in RAG architectures. The framework simulates diverse injection scenarios targeting the retrieval pipeline, integrates optional mitigation strategies such as input filtering, prompt rewriting, and retrieval-aware defenses, and automatically logs model behavior to assess attack success. By varying retrieval parameters and quantifying defense robustness across diverse attack verticals and RAG configurations, the system enables reproducible and scalable evaluation of prompt injection resilience. The results highlight strengths and weaknesses of existing defenses in RAGspecific threat models and establish a foundation for standardized benchmarking of defense mechanisms in knowledge-grounded generative AI systems.
Ankita John, Geethapriya Thamilarasu· International Conference on...· 0 citations