This work presents GraftyVul, a system that constructs vulnerable programs by grafting real-world vulnerabilities into open-source projects, and introduces a language- and context-agnostic semantic embedding that compares vulnerabilities by sink, mechanism and host-feature rather than surface code.
Abstract
Vulnerability datasets underpin a wide range of security research, including vulnerability detection, automated remediation, and secure code generation. However, existing datasets sacrifice at least one of three desirable properties: diversity (of language or vulnerability type), reproducibility/executability, or realism. We therefore present GraftyVul, a system that constructs vulnerable programs by grafting real-world vulnerabilities into open-source projects. This grounds the dataset in vulnerabilities observed in real-world contexts while harnessing known good build and test environments, enabling exploit-verification scripts to guarantee that an introduced vulnerability successfully alters a program's behaviour. Using GraftyVul, we generate 212 verified and exploitable vulnerable programs spanning five programming languages (Python, TypeScript, Java, Go, and C#) across 23 CWE categories. To evaluate fidelity, we introduce a language- and context-agnostic semantic embedding that compares vulnerabilities by sink, mechanism and host-feature rather than surface code. This approach outperforms standard code embeddings on cross-language clone and CWE classification. These embeddings demonstrate that GraftyVul samples retain a strong semantic signature to their source vulnerability. We additionally compare GraftyVul against 13 widely used datasets, where it attains competitive diversity while being the only reproducible-exploit dataset with broad language and CWE coverage. Finally, we illustrate GraftyVul's practical utility through an industrial case study evaluating a production vulnerability remediation system.
A dual-stream CodeBERT architecture is presented that addresses keyword bias —by combining pre-trained Transformer representations with a 50-dimensional hand-engineered security feature vector, supported by targeted data augmentation and two-stage adversarial fine-tuning.
Arjun Khurana, Talaya Farasat, Joachim Posegga et al.· 0 citations
Despite the dominance of Transformer-based models in software vulnerability detection, the extent to which their learned security logic generalizes across different programming languages remains a critical open question. To address this, we propose a comprehensive evaluation framework organized into three phases spanning five distinct experimental scenarios, aiming to rigorously dissect model behaviors from input sensitivity and baseline performance to cross-lingual transferability and decision transparency. We benchmark four representative models (CodeBERT, GraphCodeBERT, UniXcoder, and CodeT5+) using a curated Polyglot Injection Dataset (C/C++ and Java). Our empirical results reveal that extending input context from 128 to 512 tokens yields negligible gains, suggesting a heavy reliance on localized features. While we observe a distinct transfer asymmetry favoring C/C++ sources, the Encoder-Decoder architecture (CodeT5+) demonstrates superior cross-lingual generalization. Crucially, high structural robustness against semantic transformations, combined with quantitative XAI agreement analysis, confirms that these models capture genuine vulnerability logic. These findings suggest that the primary barrier to cross-lingual transfer is vocabulary mismatch rather than a failure in reasoning capabilities.
Nhien Huu Dinh, Chau The, Thai Hung Van et al.· International Conference on...· 0 citations
: Large language models are increasingly being deployed in safety-critical domains, yet remain vulnerable to jailbreak attacks that circumvent safety alignments. This systematic review synthesizes empirical jailbreak research published between 2024 and 2025, using a PRISMA-guided search protocol, followed by BERTopic-based topic modeling. The analysis identifies eight main jailbreak categories: optimization-based, ge-netic/evolutionary, iterative refinement, semantic/persuasion-based, decomposition, context/generation-level, visual/encoding and fuzzing attacks, and characterizes their effectiveness, efficiency, and transferability across open-source and proprietary models, including Llama-2/3, Vicuna, GPT-3.5/4, Claude, Gemini, and DeepSeek-V3. Results show that simple configuration and context-level attacks can match the near-perfect attack success rates of sophisticated white-box optimization methods on models such as Llama-2, while requiring far fewer queries and no parameter access, highlighting a gap between research focus and practical threat severity. The review further identifies five recurring vulnerability mechanisms: representation-level gaps, execution-priority manipulation, semantic fragmentation, gradient-space exploitation and persuasion susceptibility, and documents family-specific vulnerability patterns, with open-source Llama-based models consistently more exposed than safety-enhanced architectures such as Claude. Diverse methods, uneven focus on models and publication bias limit how broadly results apply. Nonetheless, the review reveals that weaknesses in safety alignment persist across successive LLM generations, urging that effective defenses must address all eight attack categories rather than isolated techniques.
Meda Račaitytė, Hélder Bastos, R. Ribeiro et al.· International Conference on...· 0 citations
VICBench enables robust evaluation of vulnerability detection approaches and shows that state-of-the-art algorithms V-SZZ and LLM4SZZ achieve only 33.3%-40.1% F1, confirming that using existing approaches still entails significant manual effort.
Jin Lu, Xuening Han, Yan Zhong et al.· 0 citations
AlIBI is presented, an automated adaptive black-box attack framework that generates and iteratively refines adversarial comments using detector reasoning and feedback and is motivated to motivate security-aware designs that carefully calibrate trust between natural-language context and program evidence.
Large language models (LLMs) demonstrate strong capabilities in code-related tasks, however their effectiveness in software vulnerability detection (SVD) remains poorly understood due to inadequate evaluation frameworks. Existing benchmarks suffer from training data contamination, isolated function evaluation without cross-component context, lack of strict vulnerable-patched pairing, and binary classification without hierarchy-aware common weakness enumeration (CWE) assessment, which prevents a reliable measurement of security reasoning versus pattern matching on leaked data. We introduce LLMKernelBench, a rigorous benchmark comprising 417 real-world Linux kernel vulnerabilities across 74 CWE types, split into a primary benchmark dataset (PBD; 314 samples, $\leq$2024) and a leakage free dataset (LFD; 103 samples, 2025 post-cutoff), with context-aware evaluation at three granularity levels and hierarchy-aware metrics quantifying semantic proximity in misclassifications. We evaluate seven LLM spanning code-specialized and general-purpose architectures. Binary vulnerability detection is near-random ($\sim\!\! 50\%$ accuracy) and strongly biased: some models label $>75\%$ of samples as vulnerable, while others mostly label them as nonvulnerable. CWE prediction is effectively unusable, with an average Top-1 accuracy of 1.4% (best: 3.3%) and a 12.3% hierarchy proximity score, providing little reliable exact or taxonomy-level signal. On the leakage-free 2025 split, binary accuracy remains near-random and robustness to multifile abstraction is model-specific rather than tied to specialization, with code-specialized and general-purpose models degrading by 2.6% and 4.5% on average, respectively. The micro-to-macro CWE-accuracy gap is larger on LFD (6.6 points) than on PBD (0.9 points), which is consistent with sensitivity to class frequency but does not identify an internal model mechanism.
Arastoo Zibaeirad, Rodrigo Pato Nogueira, Marco Vieira· IEEE Transactions on Reliabi...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
