Large language models (LLMs) are increasingly applied to software vulnerability detection, but evaluations report accuracy while ignoring inference cost and energy and under-represent open-weight models relative to proprietary systems. We benchmark eight LLMs, three frontier, and five open-weight on a stratified 1549-function subset of label-clean PrimeVul, treating cost and energy as first-class axes alongside detection quality. Cost is measured directly; energy is measured on-GPU across a concurrency sweep for three locally servable open models and FLOP-estimated with a sensitivity range for the API-served ones. Efficiency is the robust finding: open-weight models occupy the quality-efficiency Pareto frontier in every configuration tested, and no frontier model is Pareto-optimal; this is a 4-billion-parameter model matching the best frontier system’s quality at one sixty-sixth of the list price. On quality, at a matched output budget, the best open model significantly exceeds every frontier model (0.711 balanced accuracy against 0.606–0.653), although the strongest frontier system is level with the next two open models. Two findings bound the practical reading. A 125-million-parameter detector fine-tuned on PrimeVul outperforms all eight LLMs (0.765), so where in-distribution labels exist, a small task-specific model is the better instrument. It should be noted that nothing here is deployment-ready: at the natural 1:44 prevalence, precision is 2.3–8.2%.
Patrick Deininger, Wolfgang Slany· Computers· 0 citations
This work benchmarks eight LLMs generating Gherkin from three requirement corpora, scoring validity, runner acceptance, judged coverage and quality, similarity to gold standard, stability, and cost, and releases the corpora, gold standard, and prototype.
Patrick Deininger, Wolfgang Slany· Applied Informatics· 0 citations
Large language models (LLMs) increasingly repair software vulnerabilities, but most evaluations judge only similarity to a developer fix or removal of the weakness. Neither reveals whether working code was broken. We evaluate eight commercial and open LLMs on 922 JavaScript vulnerability patches, scoring neutralisation and functional preservation. Lacking tests, we score at scale with a reference-based LLM judge, calibrated against execution on a 144-patch benchmark and 254 Java-CVE patches, plus a cross-family judge. The best model fixes 23% of vulnerabilities (judge-based), and cost-efficiency inverts the accuracy ranking. Our central finding concerns the instrument: both judges flag more over-fixes than execution confirms (precision 5–10%), yet on the functional axis agree far more with each other (κ=0.75) than with execution (κ≤0.26), so judge–judge agreement measures reliability, not validity. On real Java code the over-reporting persists, while the judges’ correctness estimates diverge, leaving no single judge trustworthy. Over-fixing is genuine but, under execution, uncommon: a few percent of vulnerability-removing patches, or under 2%, excluding one artefact-prone scenario, both lower bounds. Only adequately tested execution measures the functional-preservation rate, so security-patch evaluation must run the code, use a judge only to rank models, and weigh costs. We release the harness and executable benchmark.
Patrick Deininger, Wolfgang Slany· Journal of Cybersecurity and...· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
