Comparative Performance Analysis of Manual Testing Versus Automation Testing Through ISO/IEC 25010 Quality Metrics

Abstract

Software testing is a critical component of the System Development Life Cycle (SDLC) to ensure product quality and functionality. This study conducts a comparative performance analysis between manual testing and automated testing using Katalon Studio, specifically applied to the BISA AI platform—a Massive Open Online Course (MOOC) for artificial intelligence. Following the Software Testing Life Cycle (STLC) methodology, the evaluation is based on the ISO/IEC 25010 quality framework, focusing on four key characteristics: functional suitability, performance efficiency, reliability, and usability/maintainability. A total of 72 test cases across 14 core features were executed through three iterations to ensure data consistency. The results indicate that while manual testing achieved a higher functional suitability rate of 79.16% compared to 75.00% for automation, the automated approach demonstrated significant superiority in other metrics. Automated testing recorded a performance efficiency of 85.20% (approximately 2.8 times faster than manual) and a reliability score of 94.40% by eliminating human fatigue and detecting systemic vulnerabilities such as time-outs. Overall, automated testing reached an accumulative quality score of 84.87% ("Very Good"), whereas manual testing scored 60.55% ("Good"). The findings conclude that transitioning to an automated framework is imperative for enhancing system scalability and reliability in agile development environments.