2025· Reliability and Quality of Complex Systems· 0 citations
TL;DR
The proposed hybrid approach demonstrates practical potential, but its implementation in real-world development will require improving the semantics of automatically generated contracts, expanding training sets, and creating automated error correction procedures integrated into the generation cycle.
Abstract
Background. Automated generation of formal specifications remains a key task in formal software verification to improve its reliability, since manual code annotation is labor-intensive and error-prone. The aim of the study is to evaluate the quality of generated annotations and identify practical limitations of the approach when using small-class models. Materials and methods. The experiment used the local models Qwen2.5-coder, CodeLlama, and DeepSeek-coder; syntactic checking of annotations was performed using SpotBugs, and formal verification was performed using OpenJML together with the Z3 SMT solver. Results. On a set of 11 OpenJML examples, medium-sized models demonstrated the ability to generate syntactically correct JML annotations; however, the proportion of fully verified specifications remained low. The main limitations were identified in the area of semantic accuracy of annotations, the inability to take side effects into account, and, in some cases, format mismatches between analysis tools. Conclusions. The proposed hybrid approach demonstrates practical potential, but its implementation in real-world development will require improving the semantics of automatically generated contracts, expanding training sets, and creating automated error correction procedures integrated into the generation cycle.
Experimental evaluation shows that increasing the proportion of successful verifications requires additional model training, a broader context, and the integration of mutation and candidate selection operations, while practical limitations of the proposed hybrid approach when using small-class models are identified.
P. Mel'nikov, Andrey Tyugashev· Reliability and Quality of C...· 0 citations
This work investigates whether LLM-based agents can autonomously achieve substantial performance improvements in scientific software, including mature implementations that have already been extensively optimized by human developers.
Pavlin G. Poličar, Martin Špendl, Tomaž Hočevar· arXiv.org· 0 citations
This project verified that generating robust and reliable software using agentic AI requires careful practice and human supervision and demonstrated that using agentic AI workflow for software development (with its prompts and skills) can become a specification of the code that leads to cost-effective sustainability of...
Ayesha Shafique, Barton P. Miller, E. Heymann· 0 citations
A systematic audit of technical debt in AI-generated software is presented, revealing that AI does not eliminate flaws but rather introduces a distinct "machine signature" of defects.
Yue-Cai Zhu, Nikolaos Tsantalis, Peter C. Rigby· Proceedings of the 41st IEEE...· 0 citations
Formal verification of real-world projects requires substantial manual effort, from specifying correctness properties to constructing the infrastructure needed to check them. We envision verification agents that assist developers throughout this process. Among the many tasks involved, one stands out as both foundationa...
R. Shanmuganathan, Cedric Richter, Heike Wehrheim· 0 citations
This work proposes an integrated view on the use of LLMs for EDA and establishes an LLM-enabled behavior driven hardware development workflow, introducing and defining Formal Verification Gherkin Scenarios (FV Gherkin Scenarios), unlocking CNL specifications as the foundation for formally verified hardware designs via...