Recovering vulnerability detection on realistically-distributed code

Abstract

Prior deep-learning vulnerability detectors report F1 scores (the harmonic mean of precision and recall) as high as 99% on benchmark datasets, yet collapse to the trivial all-safe classifier when evaluated on realistically-distributed code. On Real-Vul (Chakraborty et al., 2024), a corpus that simulates deployment-scale class imbalance of roughly 0.3% positive, every published model studied scores an area under the receiver operating characteristic curve (ROCAUC) near 0.50 and vulnerable-class F1 near zero. This thesis demonstrates that the collapse is not fundamental.We rebuild Real-Vul through a controlled Code Property Graph (CPG) pipeline, measure and remove a 37% content leak intrinsic to whole-codebase sampling, and train a relational graph neural network with a disciplined class-imbalance recipe: focal loss, class-aware undersampling, and fine-tuning of a GraphCodeBERT node encoder. On the deduplicated test set (n = 30,873; 518 vulnerable; 1.68% positive), the model reaches ROC-AUC 0.975 (95% CI [0.967, 0.983]) and vulnerable-class F1 0.773 (95% CI [0.746, 0.799]), where the strongest prior result on the same corpus reached F1 0.46 / AUC 0.82.Three controlled experiments locate the cause and the boundary of the result. An architectural ablation attributes the gain to the training recipe operating on a protocol-matched corpus; swapping the readout architecture leaves the metrics unchanged. A four-corpus transfer matrix shows that the result holds only within a single labeling protocol. A Java imbalancereproduction experiment shows the recipe consistently beating no-handling on the same corpus while reaching a strong detector only where the corpus carries learnable structural signal at scale. The result is a working detector on realistic data, with an account of the conditions under which it works.