OmniVul: A Holistic, Multi-Turn Conversational Benchmark for LLM-Based Vulnerability Assessment

SecVulEval: Context-Aware Benchmarking of LLMs for Vulnerability Detection

Large Language Models (LLMs) show promise for vulnerability detection, but their evaluation is limited by the lack of high-quality benchmarks. Most existing datasets rely on coarse function-level labels, overlook fine-grained vulnerability patterns, and lack critical program context such as data/control dependencies. They also suffer from data quality issues, including mislabeling and duplication, leading to unreliable evaluation and limited real-world relevance. To address these limitations, this paper introduces SecVulEval, a context-aware benchmark designed to evaluate LLMs on vulnerability detection with rich contextual information. SecVulEval focuses on real-world C/C++ vulnerabilities at the statement level. This granularity enables more precise evaluation of a model’s ability to localize and understand vulnerabilities, beyond simple binary classification at the function level. By incorporating rich contextual information, SecVulEval sets a new standard for benchmarking vulnerability detection in realistic software development scenarios. This benchmark includes 25,440 function samples covering 5,867 unique CVEs in C/C++ projects from 1999 to 2024. We evaluated state-of-the-art LLMs in both standalone and multi-agent settings. Results on our dataset indicate that current models remain far from accurately identifying vulnerable statements within a given function, although agent-based approaches provide modest but promising improvements. The best-performing Claude-3.7-Sonnet-driven agent achieves an F1-score of 23.83% for vulnerable statement detection. We believe this benchmark can serve as a foundation for advancing context-aware vulnerability detection with LLMs.

Large Language Model Vulnerabilities

Detecting Vulnerability-Inducing Commits via Multi-Stage Reasoning with LLM-Based Agents

Detecting vulnerability-inducing commits (VICs) at submission time is critical for improving the security and reliability of software systems. However, this task is highly challenging because it requires reasoning about the semantic impact of code changes from heterogeneous information sources, including code diffs, commit messages, and the surrounding contextual code. Existing approaches often struggle to fully capture these complex interactions, resulting in limited detection performance. In this paper, we propose VIC-RAGENT, an LLM-based multi-agent framework for effective and explainable vulnerability detection. VIC-RAGENT leverages multiple specialized agents to provide complementary perspectives, including structural analysis, intent understanding, and vulnerability inspection. To further improve detection reliability, the framework employs a multi-stage reasoning process that progressively refines candidate vulnerabilities through preliminary inspection, reanalysis, and a final decision stage. Experimental results on a real-world dataset across multiple LLMs demonstrate that VIC-RAGENT consistently outperforms baselines, including Direct, CoT, and CodeAgent. Compared to the strongest baseline, VIC-RAGENT achieves 1.2-1.7x higher F1-scores across different models. Overall, VIC-RAGENT offers a robust, explainable, and practical solution for detecting VICs in modern software development workflows.

Multi-dimensional Vulnerability Analysis of LM Safety Guards via Role-Goal-Format Decomposition

Current safety guard models for Large Language Models (LLMs) often fail to block complex Jailbreaking attacks because they focus on the structure of a prompt rather than its true intent. To evaluate this problem, we introduce the Role-Goal-Format (RGF) framework, which analyzes how the combination of a user’s persona, their objective, and the text format can bypass safety filters. We compared two small safety guard models: Prompt Guard 2 and Llama Guard 3. The results show that even with the same overall accuracy, they have completely opposite weaknesses. Prompt Guard 2 is easily bypassed by unusual text formats. On the other hand, Llama Guard 3 lacks semantic understanding, missing 76.7% of virus-related attacks. Furthermore, both models exhibit a common vulnerability to contextual breaking; they fail to activate safety alignment when harmful objectives are framed within authoritative and benign personas, such as Teacher, Engineer, and Investigator. These findings highlight that small safety guard models have clear limitations and require multi-layered defenses that understand both text structure and actual intent.

From Jargon to Action: Evaluating Open-Weight LLMs for Plain Language Summaries of Pentest Reports

Penetration testing reports are a critical artifact in the cybersecurity workflow, yet their technical complexity frequently limits their utility for non-specialist stakeholders involved in risk remediation decisions. This paper investigates the feasibility of using four open-weight large language models, DeepSeek-r1:32b, Qwen3.5:35b, Gemma4:31b, and GLM-4.7-flash:32b, to generate plain-language summaries of penetration testing reports. A corpus of 65 publicly available pentest reports was used for evaluation. Model outputs were assessed across four dimensions: readability, technical term density, semantic similarity to the source document, and factual correctness using LLM-as-a-judge evaluation. Two classical extractive methods, LSA and TextRank, were included as baselines. Readability analysis using seven established metrics showed that Qwen3.5 and Gemma4 produced the most accessible summaries, reducing mean Flesch Reading Ease scores from 25.2 in the originals to 49.2 and 52.1 respectively, and lowering grade-level scores from post-graduate to high-school equivalents. Results across the remaining evaluation dimensions further indicate that appropriately selected open-weight LLMs can produce accessible and factually grounded summaries of technical security documents, offering a practical alternative to proprietary solutions in privacy-sensitive deployment contexts.

VICBench: A Multi-Language Benchmark for Code Vulnerability Detection

Evaluating security vulnerability detection tools requires benchmark datasets with vulnerability-inducing commits (VICs) - the commits that first introduce vulnerabilities into codebases. VICs are essential for determining the full range of vulnerable software versions. Existing vulnerability datasets suffer from limited programming language coverage, restricted patch complexity, and narrow project scope. Through our dual annotation by human experts and an agentic workflow, we create a benchmark - VICBench - of 100 verified VICs for 100 CVEs across 88 projects in Python, Java, and C++, covering 48 CWE types. VICBench features complex real-world vulnerability fixes averaging 38.6 lines and corresponding VICs of 252.5 lines - significantly larger than prior work. Our evaluation shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort. VICBench enables robust evaluation of vulnerability detection approaches.