Reporting of external validation studies needs improvement: a systematic review of reporting in oncology prediction model studies.
TL;DR
External validation studies were substantially less common than prediction model development studies and reporting of key methodological and performance details were frequently incomplete despite the availability of TRIPOD, suggesting it may sometimes be treated as a procedural step rather than a study designed to evaluate model transportability.
Abstract
Background
External validation is essential for assessing the generalisability and transportability of a clinical prediction model. A previous review of studies published in 2010 identified substantial deficiencies in the reporting of external validations. In 2015, the Transparent Reporting of a multivariable prediction model for Individual Prognosis Or Diagnosis (TRIPOD) statement was introduced, though its impact is unclear. Although external validation is widely recommended, its prevalence in recent oncology prediction model studies is uncertain, despite the large number of models developed in this field. We aimed to examine the proportion of oncology prediction model studies including an external validation and the reporting completeness of these studies, providing a cross-sectional overview of current practice.
Methods
We searched MEDLINE (via OVID) for primary studies published between 1 June and 31 July 2023. Eligible studies evaluated multivariable prediction models in oncology using data not used for model development, including temporal or geographical split-sample approaches. Reporting completeness was assessed using the TRIPOD statement. Screening and data extraction were performed in duplicate. Findings were summarised using counts and percentages.
Results
Of 287 eligible oncology-based prediction model studies, 89 (31%) included an external validation component, with only three studies (1%) performing external validation without model development. Most validations (78/89, 88%) were conducted on newly developed models within the same study, and 32/89 (36%) used temporal or geographical split-sample approaches. Study design and participant characteristics were frequently reported, but outcome and predictor definitions were complete in only 64% and 46% of studies, respectively. Only one study reported a sample size calculation. Performance measures were reported with confidence intervals in 52/89 (58%). Calibration was assessed in 57/89 (64%), and clinical utility in 33/89 (37%). Only 22/89 (25%) explicitly described how predictions were calculated in the validation dataset. Despite validation data being clustered in 31/89 studies, heterogeneity across centres or subgroups was never examined. Open science practices were uncommon, including study registration (2.2%), protocol availability (1.1%), and code sharing (4.5%).
Discussion
In this two-month cross-sectional snapshot in 2023, external validation studies were substantially less common than prediction model development studies and reporting of key methodological and performance details were frequently incomplete despite the availability of TRIPOD. External validation was often reported without clear articulation of its objectives or contextualisation of model performance, suggesting it may sometimes be treated as a procedural step rather than a study designed to evaluate model transportability. Greater emphasis on transparently reporting external validation studies is required to enable reliable evaluation and comparison of prediction models.