Simulation of covariate and concept drift in machine learning hospital admission prediction from emergency triage
TL;DR
An Extreme Gradient Boosting model is developed using deidentified data from all presentations to a metropolitan hospital in Western Australia to demonstrate the importance of retraining to address drift for temporal model deployment.
Abstract
Machine learning models that predict hospital admission at triage may support patient flow forecasting, yet the effects of covariate drift, concept drift, and retraining on long-term performance are poorly understood. We developed an Extreme Gradient Boosting (XGBoost) model using deidentified data from all presentations to a metropolitan hospital in Western Australia. Training and validation included 2016 and 2017 presentations (n = 133,814), with rolling quarterly testing from 2018 to 2023 (n = 455,496). Two adaptive strategies were evaluated: quarterly and half-yearly retraining. Covariate drift was assessed using univariate and multivariable analyses, and reporting adhered to TRIPOD + AI and MINIMAR standards. Substantial drift was observed both between training and testing datasets and across the six-year testing period. The base model achieved a mean AUROC of 0.875 (range 0.844–0.887) and mean daily bed error of 7.42 beds (range 0.47–13.1). Retrained models demonstrated improved discrimination (mean AUROC 0.892 and 0.893) and reduced bed error (4.02 and 4.48 beds per day) for quarterly and half-yearly retraining, respectively, with similar calibration and classification performance. Covariate drift meaningfully degraded calibration but not discrimination metrics over time. Simple retraining improved discrimination and reduced calibration concept drift, underscoring the importance of retraining to address drift for temporal model deployment.