AI Security Testing Outruns Its Evidence: A Literature Survey of the State of the Field as of October 2026

Abstract

Artificial intelligence moved from an assistant for individual security-testing steps to a credible autonomous finder, exploiter and fixer of software vulnerabilities between 2023 and October 2026, but the evidence for that shift is uneven. This narrative survey synthesises research on four technical strands (AI-driven penetration testing and offensive automation; AI-assisted fuzzing and dynamic testing; AI-based vulnerability detection and automated repair, including the security of AI-generated code; and benchmarks and evaluation methodology) and two contextual strands (industry adoption and economics; risks, law and governance). Every claim is graded by evidence tier: peer-reviewed, preprint, government or organiser report, vendor claim, or secondary summary. The best organiser-scored result is the DARPA AI Cyber Challenge final (August 2025), in which seven cyber reasoning systems found 54 of 63 synthetic vulnerabilities and patched 43 at an average of about USD 152 per task. Peer-reviewed work supports a consistent design lesson: the reproducible gains come from hybrids in which a large language model supplies semantics (harnesses, specifications, taint sources, triage) and a deterministic component (fuzzer, static analyser, prover, exploit replay) supplies ground truth. By contrast, the most dramatic 2026 results, such as frontier models reported to find thousands of zero-day vulnerabilities, are vendor claims that third parties cannot yet reproduce. Independent benchmarks report far lower end-to-end autonomy than system papers, public benchmarks saturate or leak within 12 to 24 months, and about half or more of functionally correct AI-generated code remains exploitable in controlled studies. The binding constraints in 2026 are verification of findings, remediation capacity, safe operation of agents, and governance of access and disclosure, rather than discovery itself. We close with ten open problems and a statement of this survey's limitations.