Capture-the-Flag (CTF) benchmarks are widely used to assess the offensive security capabilities of autonomous language-model agents. Evaluations rely on shallow binary judgments or aggregate scores, overlooking the agent's trajectory to the flag. Consequently actual exploitation is conflated with direct flag exposure, memorized recall, external lookup, guessing, and unsupported claims, potentially overstating the agent's cybersecurity capability. We introduce CTF-ABACUS, a trace-based agent auditing framework that reconstructs each run as an evidence-grounded solve profile. By decomposing agent actions into penetration-testing phases and categorical techniques, it identifies where exploitation occurs, where the flag first appears, and whether the recovered flag is supported by demonstrated behavior. Aggregating solve profiles across agents yields challenge signatures that reveal whether success was achieved via the intended exploit or via shortcut pathways. We apply CTF-ABACUS to 1,435 CTF attempts by six frontier and open-source models on 240 challenges, yielding 2,870 solve profiles under two judge lenses. Trace-verified exploits account for only 62-87% of recovered flags across benchmarks, while shortcut recoveries follow substantially shallower trajectories. These findings shift CTF evaluation from counting recovered flags to verifying demonstrated exploitation and provide a basis for designing benchmarks that better isolate the offensive capabilities.
Kimberly Milner, Minghao Shao, Nanda Rani et al.· 0 citations
Large language models (LLMs) show promise for vulnerability discovery, yet prevailing methods often inspect code in isolation, struggle with long project contexts, and produce coarse function- or file-level predictions that offer limited guidance for targeted patches. This paper presents a controlled empirical study of what factors help LLM agents perform project-level, line-level vulnerability localization from runtime failure evidence. We introduce T2L (Trace-to-Line), a reproducible research framework that narrows repository-scale code into candidate vulnerable lines through AST-based chunking, structured diagnostic information collection, and evidence-guided refinement. To support systematic evaluation, we build T2L-ARVO, an expert-verified 50-case benchmark spanning five crash families in real-world open-source projects, and T2L-Agent, a modular reference agent for studying localization workflows under equal per-case budgets. Across ten models and controlled ablations, we find that structured diagnostic analysis, coarse-to-fine refinement, and multi-hypothesis exploration all improve trace-to-line localization. Across these workflow-level configurations, the largest observed increase in line-level localization reaches 48.9 percentage points over the corresponding baseline, while increasing reasoning budget or adjusting sampling temperature yields limited and inconsistent benefits. Overall, our study provides an empirical reference for designing LLM-agent workflows for complex project-level localization tasks that move from runtime failure evidence to precise vulnerable-line hypotheses.