Skip to content

Self-Spec Verifiable Code Generation

Sep 2026 · arXiv.org · Vol abs/2609.39568 · 0 citations · 32 references
Computer Science

TL;DR

VeriCodeBench is introduced, a benchmark for self-spec verifiable code generation, where the LLM relies solely on its own generated specification and code throughout the entire process, and CodeNova is introduced to enhance the capabilities of LLMs in self-spec verifiable code generation.

Abstract

Large language models (LLMs) may generate unreliable code on corner cases missed by testing, while formal verification can provide machine-checkable guarantees. Recently, researchers have proposed several benchmarks to evaluate the capabilities of LLMs in generating formally verifiable code, where LLMs need to formulate formal specifications, generate the corresponding code, and verify its correctness. However, existing benchmarks have two key limitations: (I) They primarily evaluate specification and code generation stage-wise, with code generation typically conditioned on an oracle specification. This setup overlooks whether strong stage-wise performance translates into end-to-end success. (II)They mainly focus on a single proof-oriented language and mathematically structured tasks, offering limited coverage of tasks common in software development. In this paper, we introduce VeriCodeBench, a benchmark for self-spec verifiable code generation, where the LLM relies solely on its own generated specification and code throughout the entire process. VeriCodeBench contains 400 language-native problems across C, Java, Rust, and Python, covering practical concerns in software development. We evaluate specification coverage, code validity, and joint problem-level success. We further introduce CodeNova to enhance the capabilities of LLMs in self-spec verifiable code generation. CodeNova makes requirements explicit through constraint-guided specification and uses verifier feedback to guide targeted implementation repairs. Experimental results reveal that self-generated specifications remain a major bottleneck, while providing more sophisticated specifications may not necessarily lead to higher verification success rates. CodeNova substantially improves performance across all evaluation metrics, enabling Claude Sonnet 5 to achieve the strongest results under the self-spec protocol.

View source

Similar papers

Preprint Oct 2026

Beyond Functional Correctness: Static Code Quality Profiling of LLM-Generated Code

Large language models (LLMs) are increasingly used for code generation, yet most evaluations still focus on whether generated code passes benchmark test suites. Functional correctness alone, however, provides only a limited view of quality: benchmark tests are often incomplete, and passing them does not guarantee that...

Jue-Zhe Huang, Tarek Mahmud, Mei-Ru Che et al. · 0 citations
Review Open access Oct 2026

Large Language Models for Verilog Generation and Verification: A Review of Cloud, Domain-Specialised and Resource-Constrained Approaches

Register-transfer-level (RTL) design is a critical but labour-intensive stage in digital hardware development. Large language models (LLMs) have recently shown the ability to convert natural-language specifications into Verilog code, thereby reducing manual design effort. This paper presents a review of LLM-based Veril...

Amol Dhote, Rajnandinee Patle, Mansi Borkute et al. · 0 citations
Book Open access Oct 2026

Beyond Benchmarks: A Case Study of LLM-Generated Verus Specification Failures on Asterinas Vostd

Large language models (LLMs) have recently shown promise in automating formal specification generation, but their effectiveness has mainly been demonstrated on self-contained benchmark tasks. This paper examines the changes that occur when LLM-generated specifications are applied to an industrial verification project....

Du-Gang Liu, Ding Wang, Chao Jiang et al. · 1 citation
#artificial intelligence Preprint Oct 2026

How Stable Are LLM-Generated Proofs Under Program Refactoring? A Stress Test for Verifiable Code Generation

RefactorProof measures unchanged-proof stability under program refactoring, not the cost of maintaining a proof after it breaks, which derives 458 Lean-certified variants from the 185 VERINA tasks that compile in the authors' pinned environment while holding the specification and evaluated proof fixed byte-for-byte.

A. Mandiga, Guo-Ming Li · 0 citations
Book Open access 2027

From Specification to Kernel Commit: Verified Code Generation on Real-World Systems

Verified code generation, in which an LLM produces both an implementation and a correctness proof, offers a promising path toward correct-by-construction software. Prior evaluations report that LLM capability remains unsatisfactory, but they test models in a bare-model setting that no longer reflects how LLMs are deplo...

Hao Sun, Ze-Nan Li, Cong Li et al. · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.