VeriCodeBench is introduced, a benchmark for self-spec verifiable code generation, where the LLM relies solely on its own generated specification and code throughout the entire process, and CodeNova is introduced to enhance the capabilities of LLMs in self-spec verifiable code generation.
Abstract
Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees. Recently, researchers have proposed several benchmarks to evaluate the capabilities of LLMs in generating formally verifiable code, where LLMs need to formulate formal specifications, generate the corresponding code, and verify its correctness. However, existing benchmarks have two key limitations: (I) They primarily evaluate specification and code generation stage-wise, with code generation typically conditioned on an oracle specification. This setup overlooks whether strong stage-wise performance translates into end-to-end success. (II)They mainly focus on a single proof-oriented language and mathematically structured tasks, offering limited coverage of tasks common in software development. In this paper, we introduce VeriCodeBench, a benchmark for self-spec verifiable code generation, where the LLM relies solely on its own generated specification and code throughout the entire process. VeriCodeBench contains 400 language-native problems across C, Java, Rust, and Python, covering practical concerns in software development. We evaluate specification coverage, code validity, and joint problem-level success. We further introduce CodeNova to enhance the capabilities of LLMs in self-spec verifiable code generation. CodeNova makes requirements explicit through constraint-guided specification and uses verifier feedback to guide targeted implementation repairs. Experimental results reveal that self-generated specifications remain a major bottleneck, while providing more sophisticated specifications may not necessarily lead to higher verification success rates. CodeNova substantially improves performance across all evaluation metrics, enabling Claude Sonnet 5 to achieve the strongest results under the self-spec protocol.
Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that...
Jue-Zhe Huang, Tarek Mahmud, Mei-Ru Che et al.· 0 citations
This work presents SpecRead, a benchmark that isolates specification comprehension from generation ability, and is automatically scorable by deterministic checks, with gray-zone cases counted wrong under the conservative main scoring.
Register-transfer-level (RTL) design is a critical but labour-intensive stage in digital hardware development. Large language models (LLMs) have recently shown the ability to convert natural-language specifications into Verilog code, thereby reducing manual design effort. This paper presents a review of LLM-based Veril...
Amol Dhote, Rajnandinee Patle, Mansi Borkute et al.· International Journal of Res...· 0 citations
Large language models (LLMs) have recently shown promise in automating formal specification generation, but their effectiveness has mainly been demonstrated on self-contained benchmark tasks. This paper examines the changes that occur when LLM-generated specifications are applied to an industrial verification project....
Du-Gang Liu, Ding Wang, Chao Jiang et al.· Proceedings of the 41st IEEE...· 1 citation
RefactorProof measures unchanged-proof stability under program refactoring, not the cost of maintaining a proof after it breaks, which derives 458 Lean-certified variants from the 185 VERINA tasks that compile in the authors' pinned environment while holding the specification and evaluated proof fixed byte-for-byte.
Verified code generation, in which an LLM produces both an implementation and a correctness proof, offers a promising path toward correct-by-construction software. Prior evaluations report that LLM capability remains unsatisfactory, but they test models in a bare-model setting that no longer reflects how LLMs are deplo...
Hao Sun, Ze-Nan Li, Cong Li et al.· Proceedings of the 32nd ACM...· 0 citations