Machine Learning for Diabetes Screening: Insights from the Behavioral Risk Factor Surveillance System and Pima Indians Diabetes Databases
TL;DR
A simple logistic regression – with excellent recall and competitive AUC – appears preferable for population screening, where transparency, robustness to reporting noise, and ease of implementation are paramount.
Abstract
Background: The increasing prevalence of Type 2 Diabetes Mellitus (T2DM), currently estimated at 462 million individuals, corresponding to 6.28% of the world’s population, highlights the critical need for effective screening strategies that leverage large-scale health surveys and clinical databases for early risk identification and intervention. Machine learning approaches applied to such data have reported near-perfect predictive accuracy such as accuracy of 0.98 on the BRFSS dataset and 0.97 on the PIDD dataset, but the reproducibility of these results under fully specified, independent pipelines remains largely untested. Objective: T2DM is a public health challenge in which early screening can significantly improve outcomes. The objectives were (i) to develop and validate machine-learning models for diabetes risk using large, survey-based data and (ii) to evaluate reproducibility of previously reported high-performance models, while identifying interpretable risk factors suitable for deployment in screening. Methods: We performed a cross-sectional retrospective analysis of the 2015 Behavioral Risk Factor Surveillance System (BRFSS) and the Pima Indians Diabetes Database (PIDD). Models included logistic regression, tree-based learners (e.g., XGBoost), and a ConvLSTM deep network. We systematically explored 2,240 configurations spanning preprocessing, scaling, class-imbalance remedies (e.g. SMOTE variants), and regularization. Interpretability relied on SHAP to quantify feature contributions. Results: Attempts to reproduce a previously published KNN recipe’s near-perfect performance demonstrated significantly lower performance on BRFSS (0.69 accuracy). A similar result was obtained in ConvLSTM architecture replication on PIDD reached ~0.79 test accuracy. In BRFSS, CatBoost achieved the top AUC (0.83), while a tuned logistic regression delivered the most screening-appropriate profile (recall 0.97, AUC 0.82) with transparent coefficients and straightforward deployment. SHAP consistently ranked self-reported general health, BMI, and age as the dominant predictors. A prespecified sensitivity analysis showed the logistic regression model’s discriminative ability was largely stable when subjective or objective self-reported variables were removed individually (ΔAUC ≈ –0.02). Larger, but acceptable, degradation was observed when all self-reported features were excluded or randomly perturbed; recall remained high across scenarios. Conclusions: There is a gap between headline results in the literature and performance achieved under fully specified and reproducible pipelines. A simple logistic regression – with excellent recall and competitive AUC – appears preferable for population screening, where transparency, robustness to reporting noise, and ease of implementation are paramount. Sensitivity checks reinforce that performance does not hinge on any single subjective domain, supporting clinical utility of logistic regression models in imperfect real-world data.