Structured ACR Bone-RADS Versus Unstructured Assessment of Bone Tumors on Radiographs: A Multicenter, Multireader Study Across Experience Levels.
TL;DR
Bone-RADS provides stable risk stratification, improves diagnostic performance and interreader agreement in early-career readers, and shifts management recommendations toward greater referral or workup of potentially malignant lesions.
Abstract
Purpose
To compare the diagnostic performance and interreader agreement of structured ACR Bone Reporting and Data System (Bone-RADS) versus unstructured assessment of bone tumors on radiographs and to assess their effects on management recommendations.
Materials And Methods
Multicenter, multireader study with a primary retrospective cohort (n = 1,423; 4 centers), external retrospective cohort (n = 354; 6 centers), and prospective cohort (n = 152; 2 of these 10 centers; written informed consent obtained from all prospective participants). After a standardized training session with practice cases, nine musculoskeletal radiologists (early career 3-5 years; midcareer 15-20 years; late career >20 years) independently performed unstructured classification and Bone-RADS 1 to 4 grading with a standardized 4-week washout. Primary end point was reader-averaged difference in area under the curve (ΔAUC) via an Obuchowski-Rockette-Hillis multireader, multicase receiver operating characteristic model; interreader agreement and management recommendation reclassification were also assessed.
Results
Overall reader-averaged AUC did not differ significantly between unstructured assessment and Bone-RADS across the primary retrospective, external retrospective, and prospective cohorts (ΔAUC: -0.0012, 0.0160, and 0.0053, respectively; all P > .05). However, early-career readers demonstrated significant AUC improvements with Bone-RADS in the primary and external cohorts ΔAUC: +0.0302 and +0.0615; both P < .001), with a similar but nonsignificant difference in the prospective cohort (ΔAUC = +0.0201; P = .572). In contrast, midcareer readers showed nonsignificant changes, and late-career readers exhibited slight performance declines. Interreader agreement increased across all cohorts (eg, in the prospective cohort, early-career Fleiss κ improved from 0.394 to 0.502). Feature-level agreement was highest for pathologic fracture but lower for margins and endosteal erosion (κ ∼0.518-0.660). Malignancy rates increased monotonically across Bone-RADS categories 1 to 4 (9.44%, 25.44%, 46.67%, and 79.67%, respectively). Among early-career readers, Bone-RADS shifted management recommendations by increasing referral or further workup recommendations for potentially malignant lesions in the primary (+8.61%), external (+15.30%), and prospective (+7.14%) cohorts, while concurrently reducing routine surveillance recommendation rates for benign lesions (-17.27%, -15.90%, and -19.51%, respectively).
Conclusion
Bone-RADS provides stable risk stratification, improves diagnostic performance and interreader agreement in early-career readers, and shifts management recommendations toward greater referral or workup of potentially malignant lesions. Its sensitivity-oriented design may increase overmanagement of some benign lesions. To mitigate this, future refinements should consider incorporating age and anatomic-site information, adopting context-aware management thresholds, and providing structured training on lower-agreement features such as margin classification and endosteal erosion.