Predicting Test Automation Failures using Machine Learning for Proactive Quality Engineering

Abstract

Test automation is vital for software release in modern times, it often results in false positives due to failed tests, which leads to the waste of precious engineer's time and, hence, loss of trust in the system. The proposed research project seeks to design a predictive model that can estimate the possibility of false positive failure before test automation. Our model was trained using the data on 12,400 test cases from 840 automated tests that were conducted during 18 months of continuous integration process. In total, there were 42 features selected as input data for training – they include data from logs, information about code changes, environmental factors, and failure history. Five machine learning algorithms were tested, XGBoost algorithm being the most successful with accuracy being 91.2%, while F1-score equalled 90.5%. Testage, code churn, and failure rate were the most important features. After deploying our tool into production CI pipeline, we noticed a reduction inthe number of test cycles of up to 34% as well as increase of defect detection speed by 60%.