Large language models (LLMs) have shown strong potential for translating natural-language (NL) requirements into PL/SQL programs, attracting increasing attention from the database community. However, existing NL-to-PL/SQL efforts primarily focus on directly generating PL/SQL from complete NL requirements. In practice, PL/SQL development involves diverse scenarios, such as from-scratch development, code modification, debugging, and optimization, and may require either direct generation or multi-turn interaction. Yet, no comprehensive benchmark evaluates multi-scenario, direct and interactive, and multi-dialect NL-to-PL/SQL development. In this paper, we present ProcArena, an execution-based benchmark covering both Direct and Interactive modes. ProcArena comprises 3,998 executable tasks over 157 databases, spanning nine development subscenarios in PostgreSQL and Oracle. We construct challenging Direct tasks through Iterative Logic Enhancement and scenario-specific adapters, and derive paired Interactive tasks through Knowledge Integration and Requirement Perturbation while preserving executable targets. We further design a controlled Solver-User Simulator protocol that allows models to clarify user intent and inspect the database environment without exposing hidden execution feedback. Evaluating seven language models, we find that the best average scores are only 62.2% and 57.8% in Direct and Interactive, respectively, demonstrating that realistic NL-to-PL/SQL development remains challenging, particularly in interactive settings.
Hang Zhang, Chao-Kun Wang, Yun Pan et al.· 0 citations
Smart contract failures can cause irreversible financial and operational losses, yet current validation workflows still rely heavily on warnings, execution traces, and manually authored tests that are difficult to scale for contract-specific logic. Existing automated tools—static analyzers, symbolic executors, and fuzzers—primarily produce issue-centered outputs such as warnings or counterexamples rather than structured, reviewable test artifacts. This paper investigates whether large language models (LLMs) can bridge this gap by generating structured draft test specifications from Solidity source code. We propose AutoTestAI, an exploratory implementation-oriented framework that combines contract preprocessing, a structured auditor-style prompt, and multilayer output purification to produce machine-readable CSV test specifications suitable for downstream review and possible test implementation. On a 27-contract main benchmark, AutoTestAI achieves 91.3% overall function-level coverage. Under the revised baseline-and-ablation framework, CSV purification improves mean contract-level coverage on the Main27 split from 0.00% (strong_raw) to 91.53% (strong_purified), while enabling preprocessing within the full pipeline improves Holdout40 mean coverage from 69.73% (autotestai_no_preprocess) to 83.46% (autotestai_full), with both coverage gains statistically significant under paired Wilcoxon testing ( $p{\lt }0.01$ ). An expanded Foundry executability-convertibility validation shows 141/141 converted tests compiled and 133/141 executed successfully. A fresh 40-contract multi-project holdout set and a model sensitivity check with gpt-4o further examine broader applicability and model dependence. The method is intended as a practical complementary layer alongside static analysis, symbolic execution, fuzzing, and expert review, rather than a production-ready auditing framework or a substitute for formal verification, exhaustive testing, or expert judgment.
Shengyu Xie, Xingxing Yang, Yun Pan et al.· IEEE Access· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.
