Prospective evaluation of a real-time AI assistant for outpatient documentation and quality control

Abstract

Outpatient electronic medical record (EMR) quality is highly variable, yet few tools exist to support real-time documentation. Here we developed SMART-ASSISTANT, an end-to-end AI system that automates EMR writing for outpatients through integrated audio transcription, structured EMR generation, quality control, and assisted diagnosis. Using abdominal pain as the exemplar, model performance was validated in simulated, retrospective, multi-reader multi-case (MRMC), and prospective cohort studies. SMART-ASSISTANT outperformed GPT-4 in identifying key clinical features (symptoms: 100.00% vs 19.20%; onset characteristics: 85.60% vs 45.60%; disease progression: 89.60% vs 45.40%; all P <0.001). In the MRMC study, physicians rated AI-assisted EMRs significantly higher than human-generated ones for completeness (4.27 vs 3.92, P <0.001) and diagnostic correlation (2.53 vs 2.33, P <0.001). In the prospective cohort, AI-generated EMRs showed higher integrity than physician-generated EMRs (3.50 vs 3.21, P =0.008), but lower factuality (3.34 vs 4.01, P <0.001), with no significant differences in normativity, readability, or logicality. SMART-ASSISTANT may support outpatient documentation by improving information completeness, but physician verification remains essential. Trial registration : Chinese Clinical Trial Registry, ChiCTR2400086606. Registered 08 July 2024, https://www.chictr.org.cn/ .