Aug 2026· 3 citations· ⚡ 1 influential· 36 references
Computer Science
TL;DR
An episode-level evaluation protocol for healthcare NLP agents is introduced, supplying a reproducible intermediate evaluation layer between static benchmarks and prospective workflow studies, with an explicit cost-sensitive treatment of missed versus unnecessary escalation.
Abstract
Large language model (LLM) agents are increasingly proposed for healthcare tasks such as clinical documentation, evidence retrieval, patient messaging, and care coordination. Yet many evaluations remain limited to static medical question answering or one-shot generation, under-representing longitudinal state, interruptions, and human handoffs. We introduce an episode-level evaluation protocol for healthcare NLP agents. The protocol separates evidence across model, agent, and simulated-workflow behavior; specifies a five-field episode schema; and defines annotation and scoring for state continuity, evidence traceability, and escalation decisions. It is instantiated as four task templates: documentation update, evidence retrieval, patient messaging, and triage handoff. The protocol does not claim to measure clinical outcomes or deployment value. Instead, it supplies a reproducible intermediate evaluation layer between static benchmarks and prospective workflow studies, with an explicit cost-sensitive treatment of missed versus unnecessary escalation.
CLAIRE (Clinical Language and Agentic Intelligence for Reasoning and Entry), a hybrid workflow that separates field-state discovery, source-to-field mapping, deterministic validation, bounded correction, escalation, and audit tracing, is developed and evaluated.
In hospital workflows, electronic health records (EHRs) are often noisy, and may not contain the evidence needed to confirm events or measurements referenced in a clinical query. Even when database retrieval succeeds, clinical agents can overlook such discrepancies and return plausible but unsupported answers. We intro...
Yi-Tong Qiao, Yan-Cheng Jin, Lei Liu et al.· arXiv.org· 0 citations
A scalable framework that automatically generates question--answer pairs from longitudinal EHR notes that supports evaluations that static benchmarks cannot, including the generation of multiple answers that reflect variation in clinician reasoning for robust performance assessment and continuously refreshing benchmark...
J. Cahoon, C. Stanwyck, Sulaiman Somani et al.· arXiv.org· 1 citation
Large language model (LLM)-based agents are increasingly being explored for healthcare tasks such as clinical decision support, care coordination and autonomous workflow execution. Beyond static pipelines, recent systems claim to self-evolve by adapting their tools, memory, reasoning, policy, context and coordination s...
ClinAgent is a conversational system based on agentic Retrieval-Augmented Generation that enables clinicians and researchers to query clinical trial information in plain language and receive grounded, up-to-date responses across multi-turn interactions, highlighting the potential of agentic AI systems to improve the ac...
Antonino Vaccarella, Riccardo Cantini, Domenico Talia et al.· 0 citations
The reported experiments characterize CLEAR-Med's configuration and scalability and the Invocation Agent's accuracy, while the formal analysis establishes the encoded-property guarantee of the complete control flow.
Erfan D. Dehkalani, S. Shankaran, Abbot R. Laptook et al.· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 19, 2026
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.