A multi-agent, persona-driven prompt generation algorithm generates realistic, multilingual prompts with minimal human review, leaving room for improvement with modern jail breaking techniques in AIR-BENCH Live, a self-evolving successor to AIR-BENCH 2024.
Abstract
Foundation-model safety benchmarks capture the AI risks of their time of publication: as models improve and governments pass new AI-safety legislation, their risk taxonomies become incomprehensive and their attack prompts become ineffective. We present AIR-BENCH Live, a self-evolving successor to AIR-BENCH 2024. An automated update pipeline monitors government regulation and classifies new policies against the current four-tier risk taxonomy, either matching them to existing categories or proposing new granular categories. Then, a multi-agent, persona-driven prompt generation algorithm generates realistic, multilingual prompts with minimal human review, leaving room for improvement with modern jail breaking techniques. This algorithm is used to overhaul legacy prompts and generate prompts for new categories. In our current version, the pipeline has expanded the benchmark from 314 to 335 granular risks, with the 21 new categories drawing from 31 truly novel policy clauses across seven jurisdictions. Evaluating 14 recent models, we find a wide safety spread (from 0.17 to 0.89 among the models judged on their own behavior), that the modernized prompts are on average 0.06 points harder than the 2024 set, with the largest drops concentrated among the most compliant models, and that most models are modestly less safe on non-English prompts. By continuously absorbing new regulation and regenerating prompts, AIR-BENCH Live is designed to evolve alongside a fast-moving field.
AI agents extend large language models (LLMs) with external tools, enabling them to perform complex tasks and translate model outputs into consequential real-world actions. Yet LLMs often become substantially less safe when deployed as agents, and the source of this degradation remains poorly understood. In this paper, we identify schema-formatted tool specifications as a primary source of agent safety degradation and show, through white-box representation analysis, that they weaken the model's internal refusal signals and contribute to unsafe tool execution. Building on this finding, we propose SafeKeep, an inference-time safeguard that decouples safety judgment from tool execution: it assesses requests using flattened textual tool specifications while retaining the original schema-formatted specifications for execution. Across two representative benchmarks and four LLMs, including both white-box and black-box models, SafeKeep increases the average refusal rate for harmful requests from 23.8% to 70.6% and reduces the average attack success rate under observation-level prompt injection from 25.6% to 2.5%. It also outperforms existing safeguards and preserves task-handling capability. We release the code and data at https://github.com/snowcatsmoking/SafeKeep .
Minghui Pan, Jiayuxuan Yang, Yuanyuan Yuan et al.· 0 citations
The shift from passive predictive models to autonomous agents capable of tool use and multi-step planning moves the AI safety landscape from prediction error to control failure: small misjudgements become irreversible actions, and risks compound across long horizons and populations of interacting systems. We present a structured review and taxonomy of open scientific problems in agentic AI safety, mapped explicitly onto the EU AI Act and the NIST AI Risk Management Framework. The corpus follows a PRISMA-ScR scoping review, assembled through anchor-based citation chaining and curated reading lists across arXiv, the major machine-learning conferences, and selected security and fairness venues, with a primary March 2026 search cut-off (extended to May 2026 during revision for a small number of high-relevance governance and agentic-safety sources), explicit eligibility criteria, and an analytical distinction between open scientific problems and deployment risks. The taxonomy identifies eight problem families spanning reinforcement-learning policies and language-model planners: goal specification, inner alignment, safe learning and robustness, scalable oversight, interpretability, tool-use security, multi-agent safety, and evaluation and assurance. Mapping these onto the two frameworks shows close alignment for some families and notable absences for others, with multi-agent safety surfacing as a regulatory gap. We add a per-family research roadmap with concrete milestones and a practitioner-facing deployment-posture triage, arguing that progress on inner alignment, interpretability for deceptive-alignment detection, and multi-agent safety would most directly reduce compliance uncertainty.
Tomáš Valenta, Ondrej Rozinek, J. Horálek· Applied Informatics· 0 citations
Large Reasoning Models (LRMs) generate intermediate reasoning traces that may contain unsafe content, even when their final responses appear safe. Guardrail models are designed to detect and block unsafe content, yet existing benchmarks for unsafe content detection focus primarily on prompts and final responses, leaving reasoning traces largely unexamined. Moreover, these benchmarks typically provide only binary safety labels, without evidence annotations that justify the judgments. To address these limitations, we introduce TRACE, an evidence-grounded safety evaluation benchmark that covers the entire LRM inference pipeline: prompts, reasoning traces, and final responses. TRACE includes prompts in two languages spanning nine risk categories and ten attack strategies. For each prompt, four LRMs generate reasoning traces and final responses, and we annotate the safety of each component and extract supporting evidence from the corresponding source text. Evaluating 18 guardrail models on TRACE reveals that safety judgment for reasoning traces is substantially more challenging than for prompts or final responses, and that current models struggle to accurately extract supporting evidence. These findings highlight the need for guardrail models that can reliably detect and precisely localize unsafe content across the LRM inference pipeline.
Zhenyu Wu, Siyu Chen, Changchun Yang et al.· 0 citations
Large language models (LLMs) are increasingly considered for environmental enforcement, but their ability to produce traceable enforcement decisions remains unclear. We introduce WuYu-EnvLE-Bench, a benchmark built from real enforcement cases, regulatory standards, and expert review. It contains 2,521 benchmark instances, 14 tasks, and 12 pollution-medium subdomains across pre-enforcement, in-enforcement, and post-enforcement workflows. Using Absolute Environmental Enforcement Score (AES) and Intelligent Enforcement Index (IEI), we evaluate open-source and closed-source LLMs across capability, response quality, and resource efficiency. Results show that LLMs perform well on rule-bounded tasks but remain unreliable in evidence-chain construction, contradiction detection, multi-source integration, and procedural judgment. Model scaling also shows diminishing returns: medium-sized models approach leading models in structured tasks, while larger models do not reliably overcome evidence-reasoning bottlenecks. WuYu-EnvLE-Bench highlights the need for evidence-grounded, rule-aware, and task-adaptive enforcement reasoning.
Ziliang Yang, Yi Zhang, K. Lin et al.· 0 citations
As AI coding agents take on increasingly complex, long-horizon software engineering tasks, existing benchmarks are rapidly saturating and their evaluation quality has come under serious scrutiny: a recent audit found that nearly 60% of unsolved SWE-bench Verified instances contain flawed tests -- either overly narrow tests that reject correct solutions or overly broad tests that check unstated requirements -- and that frontier models can verbatim reproduce gold patches from training data. Code refactoring, which requires coordinated, behavior-preserving changes across many files, offers a substantially harder and more realistic test of agent capability, yet remains underserved by current benchmarks. We introduce SWE-Bench ProMax, an expert-curated, multilingual code refactoring benchmark of 170 instances drawn from real commits across seven programming languages (Python, Java, TypeScript, Go, C, C++, and Rust). Every instance undergoes rigorous, multi-stage curation that directly addresses the quality problems identified in prior benchmarks: issue descriptions are rewritten from scratch to provide precise, unambiguous specifications, and test suites are manually reviewed to remove overly narrow and overly broad tests. Tasks with insufficient complexity or limited cross-file scope are filtered out, yielding a benchmark of challenging, large-scale refactoring tasks that average 11.4 modified files and 261.6 lines of code per instance, substantially exceeding the scale of existing benchmarks. Experiments with frontier models under two agent scaffolds show that the best model achieves only 41.2% resolve rate, confirming that SWE-Bench ProMax presents a meaningful and unsaturated challenge for current AI coding agents. Our benchmark is available at https://huggingface.co/datasets/swe-bench-promax/SWE-Bench-ProMax.
Generative Artificial Intelligence systems—particularly those built on Large Language Models (LLMs)—have become central to modern enterprise computing, yet they carry with them a class of vulnerabilities that traditional cybersecurity models were never designed to address. Decoder-only transformer architectures process system instructions and untrusted user inputs as a single undifferentiated sequence of tokens, which makes them susceptible to direct and indirect prompt injections, jailbreaking attacks, and retrieval poisoning. Existing governance frameworks—including the NIST AI Risk Management Framework, the EU AI Act, and ISO/IEC 42001—offer valuable compliance guidance at the organizational level, but they stop short of providing the kind of execution-level security blueprints that engineering teams actually need.
This paper introduces the Prompt-Level AI Risk Governance Framework (PLAIRGF), a four-phase architectural model covering Prevention, Detection, Response, and Continuous Improvement. Rather than depending on simulated metrics to evaluate the framework, we validate PLAIRGF through mathematical formalization of key risk indicators—including Attack Success Rate (ASR), Risk Severity Score (R), the Governance Readiness Index (GRI), and a newly proposed Human-in-the-Loop Alignment Coefficient (η)—combined with rigorous defensive capability mapping and step-by-step operational trace walkthroughs of representative adversarial attack scenarios. The paper concludes with a structured alignment between PLAIRGF controls and international compliance standards, offering organizations a practical, audit-ready foundation for deploying LLM-based systems securely.
Keywords: Generative AI Security, Large Language Models, Prompt Injection, AI Governance, Retrieval-Augmented Generation, Human-in-the-Loop, PLAIRGF
Dr. Abdul Majid Farooqi Dr. Abdul Majid Farooqi, Ziya Anjum Ziya Anjum· International Scientific Jou...· 0 citations