From segmentation to prediction: leveraging medical image analysis to predict immunotherapy outcomes in advanced melanoma

Abstract

Immune checkpoint inhibitors (ICIs) have improved outcomes for patients with advanced melanoma, yet up to half of patients do not benefit from treatment. This thesis examined whether routinely acquired medical imaging data, including H&E–stained histopathology and CT-derived body composition metrics, contain additional predictive information to support treatment decisions. In Part I, the predictive value of manually scored tumor-infiltrating lymphocytes (TILs) was evaluated. In the largest cohort studied to date, high TIL levels in pre-treatment metastatic samples were associated with higher response rates and longer progression-free and overall survival, independent of established clinical predictors. However, manual scoring showed considerable interobserver variability. Part II focused on AI-assisted histopathology analysis. A melanoma-specific annotated dataset was developed and retrained deep learning models on this dataset achieved detection performance for tumor cells and lymphocytes comparable to interobserver agreement. AI-detected TILs showed stronger associations with treatment outcomes than manual scoring, although discriminative performance remained modest. A foundation-model–based multiple instance learning approach further improved treatment stratification by capturing histopathological patterns beyond TIL density. Using this dataset, the PUMA Grand Challenge demonstrated that combining tissue and nuclei segmentation improves cell detection and identified intra-tumoral TILs as the most predictive feature. In Part III, CT-based body composition analysis showed that low skeletal muscle density and underweight status were associated with worse survival, while no independent association was found with immune-related toxicity. Overall, this thesis shows that AI-based analysis of diagnostic imaging data provides additional predictive information and may support more personalized treatment strategies in advanced melanoma.