Autonomous Quality Assurance of Enterprise Platforms and Products Using Agentic AI

Abstract

The growing adoption of Agentic AI and Large Language Model (LLM)-driven systems in enterprise platforms has introduced new challenges for quality assurance. Traditional testing approaches, which depend on deterministic outputs and exact validations, are often insufficient for systems whose behavior is probabilistic, context-aware, and dynamically evolving. This paper presents LLMTest-MCP, an autonomous quality assurance framework designed to address these challenges in modern enterprise environments. The framework utilizes an LLM in two complementary roles—as a test design assistant and as a scenario execution controller—enabling end-to-end automation of the testing lifecycle. By leveraging the Model Context Protocol (MCP), the system interacts directly with live browser sessions, allowing realistic and scalable validation of enterprise applications. LLMTest-MCP follows a structured six-stage pipeline, including test case ingestion, scenario generation, domain-specific test data creation, browser-based execution using Playwright via JSON-RPC, outcome evaluation through natural language assertions, and report generation compatible with enterprise tools such as JIRA Xray. To improve reliability, the framework incorporates a multi-pass JSON recovery mechanism to handle inconsistent LLM outputs, along with a deterministic fallback mode that ensures continuity in restricted environments. The framework was evaluated on a production-grade financial services platform. Results indicate a 2.76 times improvement in scenario coverage compared to manual testing, with scenario generation completed in under three minutes for a set of 21 test cases. Additionally, the system achieved a step classification accuracy of 94.1%. These findings demonstrate that Agentic AI, combined with MCP-based execution, can provide a practical and scalable approach to autonomous quality assurance in enterprise systems.