Open or Frontier? A Cost- and Energy-Aware Benchmark of Large Language Models for Software Vulnerability Detection

Abstract

Large language models (LLMs) are increasingly applied to software vulnerability detection, but evaluations report accuracy while ignoring inference cost and energy and under-represent open-weight models relative to proprietary systems. We benchmark eight LLMs, three frontier, and five open-weight on a stratified 1549-function subset of label-clean PrimeVul, treating cost and energy as first-class axes alongside detection quality. Cost is measured directly; energy is measured on-GPU across a concurrency sweep for three locally servable open models and FLOP-estimated with a sensitivity range for the API-served ones. Efficiency is the robust finding: open-weight models occupy the quality-efficiency Pareto frontier in every configuration tested, and no frontier model is Pareto-optimal; this is a 4-billion-parameter model matching the best frontier system’s quality at one sixty-sixth of the list price. On quality, at a matched output budget, the best open model significantly exceeds every frontier model (0.711 balanced accuracy against 0.606–0.653), although the strongest frontier system is level with the next two open models. Two findings bound the practical reading. A 125-million-parameter detector fine-tuned on PrimeVul outperforms all eight LLMs (0.765), so where in-distribution labels exist, a small task-specific model is the better instrument. It should be noted that nothing here is deployment-ready: at the natural 1:44 prevalence, precision is 2.3–8.2%.