TL;DR
A systematic evaluation framework for procedure-code recommendation is established and practical requirements for future systems that are accurate, review-efficient, and grounded in clinical evidence are identified.
This work proposes Knowledge-Guided Reasoning over Clinical Evidence with LLMs (KREL), a framework that leverages LLMs for clinical text understanding and reasoning while integrating external ICD coding guidelines as structured knowledge, and enables tight coupling between domain knowledge and LLM reasoning.
Background International Classification of Diseases (ICD) codes are widely used for reimbursement, health-service planning, and disease surveillance. Automated systems can identify diagnoses and supporting evidence across clinical documents. Yet coding is more than finding diagnoses. Decisions based on isolated information may not fully reflect the evolving clinical picture during admission. Provisional, superseded, or conflicting diagnoses can therefore persist in the final code set, complicating review and correction. We therefore developed and evaluated Clinico, a coding agent that organises evidence by time and clinical relationships to adjudicate diagnoses and update their status during admission. Methods We did a retrospective multicentre study with internal and external validation using inpatient records from two Shanghai hospitals. Routine coder-finalised codes served as references. Using 5,000 development stays, we refined Clinico's prompts and workflow without fine-tuning model parameters. We compared Clinico with three prespecified comparators, two trained on 80,340 stays. Clinico maintained an evidence-linked diagnostic ledger across each admission and reconciled related or conflicting claims before finalising the code set and selecting the principal diagnosis. The primary outcome was principal-diagnosis exact-match agreement; complete code-set micro-F1 was the key secondary outcome. Findings Evaluation included 5,000 internal and 4,109 external test stays. Principal-diagnosis exact-match agreement was 83.6% (95% CI 82.6-84.6) internally and 52.0% (50.5-53.5) externally; the next-highest observed estimates were 74.7% and 31.5%. Complete code-set micro-F1 was 90.6% (95% CI 90.2-91.0) internally and 42.6% (42.0-43.2) externally; the next-highest observed estimates were 90.3% and 25.5%. Interpretation In both hospital test sets, Clinico had the highest observed estimates of principal-diagnosis agreement and complete code-set micro-F1 among evaluated methods. Its longitudinal adjudication process links decisions to source evidence, allowing them to be checked against the record. Prospective studies should assess its effects in clinical workflows using independently adjudicated references. Funding None.
The International Classification of Diseases (ICD) coding system is central to reimbursement, registry construction, and secondary analysis, yet automated inpatient International Classification of Diseases, Tenth Revision, Clinical Modification (ICD-10-CM) diagnosis coding from discharge documentation remains difficult because clinical notes are long, diagnoses are multi-label, and supporting evidence is often scattered across multiple note sections. Existing systems also provide limited justification for why a code was assigned. This study develops and evaluates a Large Language Model (LLM)-based approach for accurate and evidence-grounded ICD-10-CM coding. Our framework combines three components: a two-stage Supervised Fine-Tuning (SFT) strategy that first aligns the base model with medical instructions and then specializes it for ICD coding, an ICD-aligned Reinforcement Learning (RL) algorithm with reward and advantage functions tailored to the hierarchical structure of ICD codes, and a Retrieval-Augmented Generation (RAG)-based verifier that removes predictions lacking external support. On the Medical Information Mart for Intensive Care IV (MIMIC-IV) benchmark, two-stage SFT improves Micro-F1 from 14.4% to 64.0%, ICD-specific RL further raises it to 66.7%, and the verifier reaches 67.5%, exceeding the strongest previously reported MIMIC-IV result by 8.8 points. These results show that domain adaptation, task-aligned optimization, and evidence-guided verification can improve both coding accuracy and the auditability of model outputs. The resulting system is intended as a decision-support tool for professional coders rather than a fully autonomous deployment.
AI-based automatic ICD-10 coding is a promising approach for clinical coding support, and future research should prioritize rare-label imbalance, reproducibility, explainability, and validation across diverse clinical settings.
Abstract Objective Clinical narrative provides a unique window into provider reasoning and attribution for automated diagnosis assignment, but large language models (LLMs) have traditionally not performed well at medical coding. We evaluate a reproducible method for automated diagnosis assignment using LLMs in clinical notes and compare with structured diagnoses. Materials and Methods We used GPT-OSS for prompt engineering and task segmentation to create a model that extracts ICD-10-CM diagnoses, with estimates of severity, currency, and importance, from progress notes. We assessed performance across multiple cohorts of patients aged 0-21 years. For each, 100 outpatient provider notes were selected across levels of severity, along with coded diagnoses from that visit (electronic health record [EHR]); a subset of 130 notes were subjected to clinical expert review. Results Comparison showed 18.7% exact code and 33.3% ICD-10-CM category match between EHR and LLM, but semantic similarity of 0.93 at the category level. Compared to expert review, LLM precision was 0.84 and recall 0.49 for exact matches, and 0.92 and 0.62, respectively, for category-level matching. In contrast, coded diagnoses showed slightly higher precision (0.94 for both cases) and substantially lower recall (0.27 and 0.43) versus expert review. Codes not identified by the LLM were more often rated by the reviewer as lower importance or certainty. Discussion We demonstrate a reusable approach to optimize LLMs for use in diagnosis extraction from clinical notes that can augment structured diagnoses and provide contextualizing metadata. Conclusion LLMs represent a viable and flexible approach to diagnosis code extraction from unstructured clinical notes.
Objective Translating free-text clinical trial criteria into computable code sets is a valuable standardization practice that is necessary for producing reproducible real-world evidence studies but requires standardized interpretation across multiple clinical vocabularies. Methods We developed TrialCode Agent, a hybrid-large language model (LLM)-terminology verification agent that generates, formats, verifies, and expands candidate codes from free-text clinical criteria. The system supports ICD-9-CM diagnoses and procedures, ICD-10-CM, ICD-10-PCS, LOINC, and RxNorm medication concepts. We compared Baseline, Hybrid biomedical retrieval-augmented generation (RAG), and terminology-guided Family expansion pipelines using Claude, GPT Qwen, and MedGemma on 40 criteria from 11 trial groups. Performance was evaluated against expert-built reference code sets using exact-code precision, recall, and F1. Results The optimal pipeline varied by model. Claude with Baseline achieved the highest performance (precision 0.755, recall 0.619, F1 0.680), followed by GPT-5.5 with Baseline (precision 0.569, recall 0.658, F1 0.610), Qwen with Hybrid biomedical RAG (precision 0.656, recall 0.470, F1 0.548), and MedGemma with Family expansion (precision 0.487, recall 0.316, F1 0.383). Hybrid biomedical RAG improved aggregate F1 only for Qwen but increased GPT-5.5 RxNorm F1 from 0.320 to 0.909. Macro-averaged results showed criterion-level gains despite lower micro-averaged aggregate performance. Family expansion increased recall across models but generally reduced precision. In staged verifier ablation, micro-F1 increased from 0.254 before verification to 0.505 after final verification and expansion. Existence/vocabulary checking removed 2,594 false-positive codes, and acceptance filtering removed 952 additional false-positive codes before controlled expansion. Conclusions Combining LLM-based clinical interpretation with deterministic terminology verification produces auditable, database-ready code sets, but retrieval and broad family expansion do not consistently improve exact-code performance. Retrieval was particularly useful for RxNorm mapping, whereas overly broad or incomplete candidate generation remained the main source of error. Deterministic verification improves code validity and query readiness but cannot replace accurate clinical interpretation.
We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.
