Sep 2026· 2026 International Russian Automation Conference (RusAutoCon)· pp. 741-745· 0 citations· 21 references
Abstract
This paper addresses stochastic variance in the decisions of large language models (LLMs) during binary classification of software vulnerabilities, using the open-source model Qwen3-8B. We evaluate the self-consistency method on a balanced Python dataset built for this purpose. The method's effectiveness depends strongly on the aggregation strategy, the number of passes K, and the model's error pattern. Under a selection rule that favors overall accuracy, Majority voting with 11 passes increased accuracy from 0.5422 to 0.5904 and F1 from 0.5615 to 0.6016. The improvement in paired correctness was statistically significant by McNemar's test, though the reduction in missed vulnerabilities (false negative rate, FNR) was modest under this setting. In contrast, two alternative strategies - Threshold voting and OR-voting - produced a larger FNR reduction at the cost of a predictable drop in accuracy. These results support flexible control over the precision-recall trade-off for security defect detection in DevSecOps workflows.
Large language models can triage malware through direct verdicts or behavioral claims scored by an external policy. We present MARS, a malware triage framework, and compare direct classification with single-pass claim scoring using the same evidence collector and identical static evidence bundles for each model. The ev...
It is argued that compile rate is a scientifically unreliable metric for single-function vulnerability repair, and this is supported with five controlled experiments over 203 vulnerable functions from Big-Vul, three open-source code LLMs (350M to 6.7B parameters), and three prompting strategies.
Om Nepal, Sushant Aryal, Oluseyi Olukola et al.· arXiv.org· 0 citations
This work evaluates Jev, a decision model that instead selects directly from a declared set of candidates and returns a probability for each, against six open-weight autoregressive models and a frontier proprietary model, and builds JevVibe, a diagnosis-guided repair agent that uses predicted CWE labels to repair code...
Arshak Rezvani, Sasha Behrouzi, Ahmad Sadeghi· 0 citations
Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness, leaving no single judge trustworthy.
Patrick Deininger, Wolfgang Slany· Journal of Cybersecurity and...· 1 citation
Large language models (LLMs) demonstrate strong capabilities in code-related tasks, however their effectiveness in software vulnerability detection (SVD) remains poorly understood due to inadequate evaluation frameworks. Existing benchmarks suffer from training data contamination, isolated function evaluation without c...
Arastoo Zibaeirad, Rodrigo Pato Nogueira, Marco Vieira· IEEE Transactions on Reliabi...· 0 citations
Language models are increasingly evaluated as vulnerability detectors, and scores reported for similar models differ widely between papers. We measured how much of that difference evaluation protocol accounts for, with model outputs held fixed. In a paired test, a model must flag a vulnerable function and clear its ver...
Maciej K. Cichoń, Bartłomiej Dmitruk· 1 citation
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
