Deep learning prediction of radiation-induced xerostomia and personalized outcome-driven treatment planning in head and neck cancer

Abstract

Objective. Radiation therapy (RT) is critical in head and neck cancer (HNC) treatment but often causes radiation-induced toxicities, such as xerostomia (RIX). While deep learning (DL)-based models show promise in predicting these toxicities, their black-box nature hinders clinical applications. This study aims to develop a robust DL model to predict RIX 12 months after RT and to leverage model interpretability techniques, specifically class activation maps (CAMs) and voxel-wise dose gradient maps (GMs), to guide personalized treatment plan optimization. Approach. A 3D ResNet-based model was trained using planning CTs, dose volumes, and salivary gland contours obtained from a retrospective cohort of 839 HNC patients. To address anatomical variations, we normalized each patient’s volumes to a common reference frame using atlas normalization. To improve spatial correspondence between deep features and patient anatomy, we integrated blur pooling and adaptive average pooling, mitigating downsampling-induced voxel shifts. Model interpretability was achieved using Grad-CAM++ and GMs. Personalized plan optimization was performed on predicted RIX-positive cases by generating avoidance contours from both CAMs and GMs to guide dose reduction. Main results. The proposed model was tested on 30 independent test cases. It achieved an AUC of 0.77 with balanced sensitivity (0.71) and specificity (0.83). Among 9 predicted RIX-positive cases, GM-guided optimization converted the predictions to RIX-negative in 7 cases (77.8%), compared with 6 cases (66.7%) for CAM-guided optimization, and achieved a greater reduction in mean model-predicted RIX probability (26% versus 20%). Significance. The proposed atlas-normalized model achieved robust discrimination, and a 9-case plan-refinement analysis showed that CAM- and GM-derived spatial information could be translated into clinically constrained avoidance objectives that reduced the model-predicted RIX probability while preserving specified target and organs at risk dose requirements. These results demonstrate the feasibility of the proposed strategy for guiding clinical interventions.