Abstract B040: Learning multimodal breast cancer phenotypes for precision oncology

Abstract

Screening mammography is the cornerstone of breast cancer detection and is routinely performed before treatment initiation, providing a unique opportunity for early risk stratification. Beyond tumor detection, screening mammograms contain imaging biomarkers that reflect underlying tumor biology and the surrounding breast microenvironment. However, treatment response and long-term outcomes are driven by complex interactions between imaging phenotypes and patient-specific clinical characteristics that are not fully captured by mammography or conventional clinicopathologic staging alone. Integrating pre-treatment mammograms with longitudinal clinical data enables comprehensive patient representations that can inform therapeutic decision-making before neoadjuvant therapy (NAC) begins. We developed a multimodal framework to learn comprehensive patient phenotypes by integrating pre-treatment mammograms with longitudinal unstructured multispecialty clinical notes. Mammograms were encoded using MammoFM, a federated breast imaging foundation model pre-trained on large-scale, multi-institutional datasets, with the four standard views (CC and MLO) fused through a multi-view attention mechanism. Longitudinal peri-diagnostic EHR notes (3 months before to 2 months after diagnosis) were processed using a large language model (LLM) to generate semantic clinical embeddings from key patient characteristics. Imaging and clinical embeddings were then integrated through a cross-modal fusion transformer to learn unified multimodal patient phenotypes that capture complementary imaging and clinical information. These transferable phenotypic representations were subsequently leveraged for downstream prediction of pathological complete response (pCR) following NAC and long-term distant recurrence risk. The multimodal phenotyping framework was developed using an internal cohort of 1,064 breast cancer patients from Mayo Clinic (AZ, MN, and FL) and evaluated on an independent held-out test set (n = 35). The learned multimodal patient representations consistently outperformed unimodal approaches across downstream prediction tasks. For long-term recurrence prediction, the multimodal phenotype achieved 85.71% balanced accuracy and an AUROC of 0.760, compared with 71.43% accuracy for mammography-only and 78.57% for clinical-only models. For prediction of pCR, the multimodal phenotype achieved 64.29% accuracy, outperforming mammography-only (35.71%) and clinical-only (46.43%) models. We developed a transferable multimodal patient phenotyping framework that integrates pre-treatment mammography with longitudinal clinical documentation to learn comprehensive representations of breast cancer patients. These multimodal phenotypes consistently outperformed imaging- or clinical-only representations for predicting pCR and long-term recurrence, demonstrating their potential as a generalizable foundation for precision oncology, risk stratification, and personalized treatment planning. Xiaoyang Wei, Madhu Sikha, Zhemin Zhang, Judy W. Gichoya, Imon Banerjee. Learning multimodal breast cancer phenotypes for precision oncology [abstract]. In: Proceedings of the 19th AACR Conference on the Science of Cancer Health Disparities; 2026 Oct 18-21; Atlanta, GA. Philadelphia (PA): AACR; Cancer Epidemiol Biomarkers Prev 2026;35(10 Suppl):Abstract nr B040.