A Comparative Analysis of Machine Learning Algorithms for the Early Prediction of Diabetes with an Evaluation of Class-Imbalance Handling
TL;DR
The study comes to the conclusion that headline accuracy is an unreliable guide in imbalanced medical prediction, that imbalance handling can change a model's practical usefulness, and that this benefit is strongly algorithm-dependent, meaning that the decision to resample should be based on the algorithm and the screening priorities rather than being applied consistently.
Abstract
Although early detection of diabetes significantly lowers its harm, machine learning models for screening are often assessed based on overall accuracy, a metric that is deceptive given the high-class imbalance that characterizes clinical data. In this work, we compared and evaluated the performance of five algorithms: logistic regression, naive Bayes, support vector machine (SVM), decision tree, and extreme gradient boosting (XGBoost) to early predict diabetes in many patients. Diabetes affected about 13.9% of the 253,680 records that were looked at in the 2015 Behavioral Risk Factor Surveillance Systems Diabetes Health Indicators dataset and to deal with this problem, people followed a process called CRISP-DM and they also used something called the Synthetic Minority Oversampling Technique or SMOTE for short. They used these things to retrain each model with the data, which was not balanced. Each model was then looked at using a few different measures, including the F1-score, accuracy, precision, recall and the area, under the ROC curve to see how well each Diabetes model was working. The Diabetes models were evaluated to see how well they were doing. Ten-fold cross-validation and a held-out test set were both used to validate the results.In the absence of imbalance handling, most models diagnosed less than one in five cases of diabetes with an accuracy of roughly 86%. While the decision tree improved somewhat and XGBoost was essentially unaffected, applying SMOTE increased the recall of the linear and probabilistic models from below 0.17 to above 0.76 at the expense of accuracy and precision. The best predictors were found to be high blood pressure, overall health, and high cholesterol. The study comes to the conclusion that headline accuracy is an unreliable guide in imbalanced medical prediction, that imbalance handling can change a model's practical usefulness, and that this benefit is strongly algorithm-dependent, meaning that the decision to resample should be based on the algorithm and the screening priorities rather than being applied consistently.