Retrospective evaluation of variant forecasting models
TL;DR
There is a potential disconnect between the performance of models vis-a-vis traditional forecasting scores and their performance in answering the questions that variant forecasts are most often used for.
Abstract
Variant forecasting is potentially useful for predicting waves of infection and for informing medical countermeasure distribution. To avoid confusion, variant forecasters should distinguish between (1) genomic sequences, which are the data, (2) the taxa (e.g., evolutionary clades or lineages) to which sequences are assigned, and (3) the modeled units formed by aggregating taxa as motivated by real-life epidemiology and practical modeling constraints. Ensembling may be ineffective for variant forecasting because there are few extant classes of variant forecasting models. Rather than rely on a diversity of model types, modelers should consider post hoc model comparison to interrogate and improve the small number of extant models. Most variant forecasting models treat the compositional prevalence of pathogen variants, not the per capita pathogen infection prevalence. This separation is acceptable for the moment but substantially reduces the utility of the resulting forecasts by decoupling variant dynamics from waves of infection. Variant forecasts are typically used to answer decision-oriented questions, like whether a variant will become dominant, while extant scoring metrics focus on a forecast’s ability to make precise estimates day-by-day. Thus, there is a potential disconnect between the performance of models vis-a-vis traditional forecasting scores and their performance in answering the questions that variant forecasts are most often used for.