Skip to content

Evaluation of large language models in api testing using open API documentation

TL;DR

This study evaluates the effectiveness of multiple state-of-the-art LLMs in generating executable REST API test cases from OpenAPI documentation and proposes a generalized prompt template that balances reasoning quality and practical usability for exploratory API testing.

Abstract

Conventional automated REST API testing approaches often depend on rule-based logic, extensive configuration, or source-code access, which limits their adaptability in rapidly evolving development environments. Recent advances in Large Language Models (LLMs) offer new possibilities for automating API test generation through their ability to interpret structured documentation such as OpenAPI specifications. However, systematic empirical evaluation of LLM performance in API testing remains limited. This study evaluates the effectiveness of multiple state-of-the-art LLMs in generating executable REST API test cases from OpenAPI documentation. Four LLMs are evaluated using four prompting strategies: zero-shot, one-shot, few-shot, and chain-of-thought prompting. Thirteen public REST APIs from diverse domains and varying levels of complexity are selected to construct a representative evaluation dataset. Model performance is measured using a multi-dimensional evaluation framework, including coverage metrics, an oracle quality metric, and a test validity metric. Descriptive analysis is complemented by non-parametric statistical methods, including the Kruskal-Wallis H test, Dunn’s post-hoc test, Spearman’s rank correlation, and the Wilcoxon signed-rank test. The results indicate that LLMs are capable of generating valid and executable API test suites directly from OpenAPI specifications, with statistically significant performance differences across models and prompting strategies. Based on these findings, the study also proposes a generalized prompt template that balances reasoning quality and practical usability for exploratory API testing. The findings demonstrate the practical feasibility of LLM-driven API testing and provide empirical guidance for model selection and prompt design for exploratory testing.

View source

Similar papers

#large language models Open access Oct 2026

On the Evaluation of Large Language Models in Unit Test Evolution (Experience Paper)

Large language models (LLMs) have recently shown promising potential in automating unit test evolution for evolving software systems. However, the effectiveness of LLMs in unit test evolution remains insufficiently understood, particularly with respect to prompt design choices, in-context learning (ICL) strategies, and...

Wei-Chang Liu, Jun-Wei Zhang, Yu-Qing Niu et al. · 1 citation
#software testing Preprint Sep 2026

How effective are traditional test criteria at detecting bugs in large language models generated code?

An empirical study involving 5 Large Language Models and 4 benchmarks evaluates the effectiveness and efficiency of 3 widely used adequacy criteria: statement coverage, branch coverage, and mutation testing, finding that mutation testing only marginally outperforms traditional coverage criteria in both triggering and d...

Asma Hamidi, Michael Konstantinou, R. Degiovanni et al. · 0 citations
Sep 2026

Evaluating the effectiveness of class-level LLM-generated test suites in Python

Reliable assessment of LLM-generated tests should treat executability as a gate and combine coverage with mutation testing and structural quality indicators, and in practice, model selection should precede prompt tuning.

Bilal I. Al-Ahmad, M. Harshvardhan, Khaled El-Fakih et al. · 0 citations
#software testing Book Open access Oct 2026

Revisiting LLMs on New Feature Implementation in Real-World Software Development Practices

A systematic evaluation of the capability of large language models (LLMs) in new feature implementation, with a task setup strictly aligned with real-world software development practices, shows that new feature implementation of LLMs adhering to real-world world software development practices remains challenges.

J. Li, Ting-Xuan Huang, Tian Zhao et al. · 0 citations
#software testing Preprint Aug 2026

An Empirical Evaluation of Using Large Language Models for Automated Model-Based Test Generation

This paper presents an empirical evaluation of Large Language Models (LLMs) for automated model-based test generation, compared with a state-of-the-art model-based testing tool (GraphWalker) and its built-in algorithms (random and quick random for edge and vertex coverage settings).

H. Şanli, Onur Kilinççeker, Cihat Çetinkaya · 0 citations
Open access

Evaluation and Distillation of Source Code Generation Tasks by Large Language Models

Two novel contributions are introduced: CodeEval and CodeQual, an open-source execution framework that provides researchers with a ready-to-use evaluation pipeline for evaluating and improving LLMs in software engineering contexts, encompassing both functional correctness assessment and subjective code quality evaluati...

Danny Brahman · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.