PD08.07. Multimodal AI-Driven Postoperative Surveillance for Oesophageal Cancer: Automated Patient-Reported Outcome Assessment and Early Complication Warning in a Prospective Study

Abstract

Esophageal Cancer: Surgical Treatment of Esophageal Cancer – early outcomes and complications Postoperative complications following oesophagectomy occur in 25–60% of patients, yet traditional patient-reported outcome (PRO) assessment relies on standardised questionnaires with limited reliability, particularly among elderly patients with low health literacy. We aimed to develop and externally validate a multimodal artificial intelligence (AI) system that automates PRO assessment from natural patient conversations and provides early warning of major postoperative complications. We conducted a prospective observational study at a tertiary cancer center, enrolling consecutive patients undergoing esophagectomy.The development cohort (Centre 1, n=196) and the temporally independent external validation cohort (Centre 2, n=103) were recruited sequentially. Using the validated Patient Symptom Assessment for Oesophageal Cancer (PSA-ESO) instrument, we collected trimodal recordings (video, audio, text) at up to 25 timepoints per patient, yielding 6,813 evaluable assessments. We fine-tuned the Qwen2.5-Omni-7B multimodal large language model with LoRA adaptation for two tasks: automated PRO symptom grading (Task A) and Temporal Transformer-based early warning of Clavien-Dindo grade II or higher complications (Task B). In a prospective implementation substudy (n=62), we evaluated the clinical impact of real-time alerts on time-to-intervention. In the external validation cohort, the trimodal PRO assessment achieved a weighted kappa of 0.801 (95% CI 0.73–0.87) and ICC of 0.858 against expert consensus, significantly outperforming audio-text bimodal (kappa 0.754, p<0.0001) and text-only (kappa 0.689, p<0.0001) configurations. The early warning model achieved an AUROC of 0.873 (95% CI 0.82–0.93) with a mean detection lead time of 28.7 hours before clinical diagnosis. The system detected 76.1% of symptom under-reporting cases. In the implementation substudy, real-time alerts reduced median time-to-intervention from 14.2 hours to 6.8 hours (p=0.003) and were associated with shorter ICU stays (3.1 vs 5.4 days, p=0.028). An end-to-end multimodal AI system can accurately automate PRO assessment from natural patient conversations and provide clinically meaningful early warning for postoperative complications, with external validation confirming generalisability across cohorts. This conversation-based approach represents a paradigm shift from questionnaire-based PRO evaluation in surgical oncology, with particular relevance for populations with limited health literacy.