This study evaluates the effectiveness of multiple state-of-the-art LLMs in generating executable REST API test cases from OpenAPI documentation and proposes a generalized prompt template that balances reasoning quality and practical usability for exploratory API testing.
Abstract
Conventional automated REST API testing approaches often depend on rule-based logic, extensive configuration, or source-code access, which limits their adaptability in rapidly evolving development environments. Recent advances in Large Language Models (LLMs) offer new possibilities for automating API test generation through their ability to interpret structured documentation such as OpenAPI specifications. However, systematic empirical evaluation of LLM performance in API testing remains limited. This study evaluates the effectiveness of multiple state-of-the-art LLMs in generating executable REST API test cases from OpenAPI documentation. Four LLMs are evaluated using four prompting strategies: zero-shot, one-shot, few-shot, and chain-of-thought prompting. Thirteen public REST APIs from diverse domains and varying levels of complexity are selected to construct a representative evaluation dataset. Model performance is measured using a multi-dimensional evaluation framework, including coverage metrics, an oracle quality metric, and a test validity metric. Descriptive analysis is complemented by non-parametric statistical methods, including the Kruskal-Wallis H test, Dunn’s post-hoc test, Spearman’s rank correlation, and the Wilcoxon signed-rank test. The results indicate that LLMs are capable of generating valid and executable API test suites directly from OpenAPI specifications, with statistically significant performance differences across models and prompting strategies. Based on these findings, the study also proposes a generalized prompt template that balances reasoning quality and practical usability for exploratory API testing. The findings demonstrate the practical feasibility of LLM-driven API testing and provide empirical guidance for model selection and prompt design for exploratory testing.
Large language models (LLMs) have recently shown promising potential in automating unit test evolution for evolving software systems. However, the effectiveness of LLMs in unit test evolution remains insufficiently understood, particularly with respect to prompt design choices, in-context learning (ICL) strategies, and...
Wei-Chang Liu, Jun-Wei Zhang, Yu-Qing Niu et al.· Proceedings of the ACM on So...· 1 citation
An empirical study involving 5 Large Language Models and 4 benchmarks evaluates the effectiveness and efficiency of 3 widely used adequacy criteria: statement coverage, branch coverage, and mutation testing, finding that mutation testing only marginally outperforms traditional coverage criteria in both triggering and d...
Asma Hamidi, Michael Konstantinou, R. Degiovanni et al.· 0 citations
Reliable assessment of LLM-generated tests should treat executability as a gate and combine coverage with mutation testing and structural quality indicators, and in practice, model selection should precede prompt tuning.
Bilal I. Al-Ahmad, M. Harshvardhan, Khaled El-Fakih et al.· arXiv.org· 0 citations
A systematic evaluation of the capability of large language models (LLMs) in new feature implementation, with a task setup strictly aligned with real-world software development practices, shows that new feature implementation of LLMs adhering to real-world world software development practices remains challenges.
J. Li, Ting-Xuan Huang, Tian Zhao et al.· Proceedings of the 41st IEEE...· 0 citations
This paper presents an empirical evaluation of Large Language Models (LLMs) for automated model-based test generation, compared with a state-of-the-art model-based testing tool (GraphWalker) and its built-in algorithms (random and quick random for edge and vertex coverage settings).
H. Şanli, Onur Kilinççeker, Cihat Çetinkaya· 0 citations
Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluati...
Danny Brahman· 0 citations
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.