Quantifying the Horizon-Reliability-Reachability Trade-off in Longitudinal Learning Analytics

Abstract

Early-warning systems in Learning Analytics are usually evaluated on how accurately a model ranks students by risk at a single chosen point in the course. This paper argues that question is incomplete, and quantifies what it omits using the Open University Learning Analytics Dataset (OULAD, 32,314 enrollments, 7 modules, 22 presentations). At five prediction horizons (weeks 1-4 and 6), we define a landmark-style risk set R_t containing only students not yet withdrawn, so a model is never credited with predicting a withdrawal that already happened. Reliability rises with horizon (best-model AUC 0.741 at week 1 to 0.808 at week 6, all consecutive gains significant by paired DeLong test, p<1e-5), but reachability -- the share of the population still in R_t -- falls over the same period, from 89.5% to 82.9%, and the students who exit are disproportionately those who would have been flagged (future-risk prevalence within R_t falls from 47.2% to 43.0%). XGBoost's advantage over logistic regression is present from the earliest horizon (+0.008 AUC, p<0.001) but amplifies roughly 2.4x by week 6 (+0.019 AUC, p<1e-10), surviving leakage-proof hyperparameter tuning. Under strict walk-forward temporal holdout, the horizon-reliability pattern generalizes in 6 of 7 modules, while conventional random-split evaluation overstates future-cohort generalization by a mean of 2.2 AUC points. These results reframe the central question for early-warning deployment: not only how reliable a prediction is, but how many students remain reachable by the time it is reliable enough to act on.