Jun 2026· arXiv.org· Vol abs/2606.29876· 0 citations· 12 references
Computer ScienceBiology
Abstract
Modern large language models (LLMs) reach 60-70% diagnostic accuracy on complex clinical case benchmarks, but accuracy alone cannot distinguish stable clinically-grounded reasoning from pattern matching. We introduce clinical reasoning graphs, structured graph representations extracted from free-text LLM diagnostic traces using a domain-grounded ontology with 5 node types and 7 edge types. We apply this pipeline to 750 traces from five LLMs across 50 New England Journal of Medicine Clinicopathological Conference cases and three prompt conditions, and test whether diagnostic traces show stable structured reasoning patterns, or diagnostic schemas, for clinically similar cases. We operationalize this as higher graph similarity among clinically similar cases than among clinically dissimilar ones. Across 15 model-condition comparisons, within-cluster and between-cluster composite similarity are nearly equal, and no comparison survives multiple-testing correction; a component-level analysis finds any residual content signal far below schema scale. Graph similarity is also nearly identical for pairs of models that are both correct (0.488) and both incorrect (0.484), suggesting that graph structure captures a dimension not reflected in diagnostic accuracy. Structured reflection prompting increases explicit discriminating-feature analysis within traces (+33%) but does not increase cross-case consistency. These results show diagnostic competence without schema-scale reasoning consistency, and indicate that final-answer accuracy should be complemented by process-level evaluation. We release the ontology, extraction pipeline, validation protocol, and the extracted reasoning graphs and similarity artifacts as resources for structured evaluation of LLM clinical reasoning.
A single LLM verifier lacks sufficient reliability to serve as a stand-alone judge of clinical reasoning at scale, and that structured human oversight remains essential.
Hyunjung Byun, Dahyoun Lee, Munyoung Jung et al.· Journal of medical systems· 1 citation· ⚡1
Clinical records contain rich evidence about patient state, but converting that evidence into reliable, structured knowledge graphs remains difficult because extraction errors, ontology mismatch, missing relations, and temporal ambiguity can propagate into downstream systems. We propose a clinical knowledge graph construction and refinement framework that combines multi-agent relation proposal, ontology-aware normalization, deterministic evidence scoring, and JEPA-based latent refinement. Rather than treating a clinical knowledge graph as a static extraction artifact, we treat it as a predictive patient-state representation. For each admission, the system constructs an evidence-scored graph from structured MIMIC-IV records and inferred clinical cross-links, then learns to recover held-out clinical relations from the observed graph context. We evaluate the refiner with leakage-free leave-one-out edge recovery (MRR and Hits@k) and held-out batch-mask evaluation (AUC and MRR). To isolate the contribution of discharge-note context, we compare a note-embedding-free configuration with a note-augmented configuration that injects real discharge-note representations only into note-grounded entities. Under the same cohort and evaluation protocol, entity-grounded note injection improves overall leave-one-out MRR by 31% relative improvement.
Kushagra Yadav, N. Prabhath, Amit Lamba et al.· 0 citations
ABSTRACT Background Clinical reasoning in contemporary practice often involves ill‐structured, poorly defined problems that span the biopsychosocial domain and require reasoning under high relational complexity. Despite sustained scholarly attention to clinical reasoning, few models are both theoretically grounded and usable for clinicians who face this complexity. At the same time, the World Health Organization Family of International Classifications (WHO‐FIC) offers a rich but complex ontology that was not designed primarily to support clinical reasoning in practice. Aim To develop a two‐stage WHO‐FIC‐based ontological model of clinical reasoning that is conceptually coherent yet remains cognitively tractable in the face of complexity. Methods A conceptual analysis was undertaken. It integrated three strands: (1) WHO‐FIC classification theory, including recent work on harmonisation; (2) empirical and theoretical literature on clinical reasoning; and (3) cognitive theories of bounded rationality, fast‐and‐frugal heuristics, relational complexity, and framing. These strands were used to derive design constraints for a clinically usable ontology. They were then synthesised into a two‐stage, graph‐based model designed to manage the complexity‐coherence trade‐off in clinical reasoning. Results Stage 1 introduces a parsimonious triad—Body, Activity, and Environment—represented as a complete three‐node graph with bidirectional relations. This triad provides an etiologically neutral and cognitively economical starting point for framing complex clinical problems. Stage 2 introduces three emergent constructs—Health Condition, Participation, and Intervention—derived compositionally from Stage 1 relations and aligned with ICD‐11, ICF, and ICHI, respectively. The model supports iterative movement between stages through composition and decomposition. It incrementally increases complexity while keeping relational load within cognitively feasible bounds and preserves the value of diagnosis while reducing its tendency to dominate problem framing. A worked case illustrates how the ontology supports reasoning about complex, time‐dependent problems through repeated movement between stages. Conclusions The proposed ontology addresses key ambiguities within WHO‐FIC, including the relationship between Activity and Participation and the perceived hierarchical privileging of diagnosis. It offers a more coherent and cognitively usable framework while respecting limits on human reasoning and managing the complexity‐coherence trade‐off. The model provides a theoretically grounded heuristic scaffold for clinicians and educators who work with complexity. It also contributes to clinical reasoning literature by emphasising reasoning about concepts, not only cases, with implications for interprofessional education and practice.
David Kellett· Journal of Evaluation In Cli...· 0 citations
Background: Clinical terminology pipelines must first extract candidate spans from narrative notes and then determine whether those spans map to existing concepts or warrant further review. Evaluation is difficult because span boundaries vary between annotators and because downstream decisions depend on the terminology evidence retrieved for each span. Objective: We evaluated clinical concept extraction, terminology linking across controlled evidence conditions, and ontology-extension triage for terms that remained unmatched after initial terminology screening. Methods: We conducted 3 complementary pilot evaluations that used distinct units of analysis and were analyzed separately. Study 1 compared 5 automated extraction pipelines and a union-merge analysis with 2 human annotation sets in 66 deidentified clinical notes from 3 health systems. Agreement was evaluated by exact string matching and BGE-large-en-v1.5 embedding matching. Study 2 evaluated 56 clinical spans, including 28 with reference Unified Medical Language System concepts and 28 adjudicated as unsuitable for ontology extension, under complete retrieval, matched-concept masking, and large language model-only inference, yielding 168 span-condition outputs. The graph retrieval pipeline used BGE-large-en-v1.5 embeddings, and the decision model was Gemma 3 27B. Study 3 applied full vector retrieval to 84 terms previously not matched in either UMLS or BioPortal. Results: In Study 1, interannotator exact-match F1 was 0.29 and embedding-match F1 was 0.75. Automated exact-match F1 scores ranged from 0.07 to 0.17; embedding-match F1 was highest for MedGemma (0.55), followed by Gemma (0.53), sci_md and SciBERT (each 0.43), and Llama 3.3 (0.32). In Study 2, complete retrieval returned a reference-matched link for 28/28 known-concept spans (100%; 95% CI, 87.9%-100%). Masking assigned POSSIBLE_CANDIDATES to all 28; large language model-only inference assigned POSSIBLE_CANDIDATES to 25/28 (89.3%) and LINKED to 3/28 (10.7%). Across the 3 evidence conditions, the same 12/28 unsuitable-extension spans were classified as NOT_MEANINGFUL (42.9%) and the same 16/28 as POSSIBLE_CANDIDATES (57.1%). In Study 3, the pipeline assigned PLAUSIBLE_EXISTING_CONCEPT to all 84 terms, none was flagged for extension, and top-candidate similarity averaged 0.914 (SD 0.027); extension status was not independently adjudicated. Conclusions: Measured extraction performance varied substantially by matching definition, whereas exact-link decisions varied with the availability of matched terminology evidence. In the follow-up sample, initial nonmatching did not establish ontology novelty: after semantic retrieval, the pipeline classified all 84 terms as plausible existing concepts and proposed none for extension. These findings support separate evaluation of extraction, retrieval, evidence-grounded linking, and extension candidacy.
Pathology reports anchor cancer diagnosis and staging, yet their narrative structure limits reliable translation into structured, machine-actionable knowledge, creating a bottleneck between expert interpretation and scalable clinical intelligence. Despite decades of clinical natural language processing (NLP) research, pathology text remains among the most complex and consequential sources of medical data to operationalize at scale. Large language models (LLMs) offer new approaches for reading, extracting, and interpreting these reports. We synthesize current LLM work in cancer pathology using a four-level capability framework across the pathology report data lifecycle: (level 1) text preparation and quality checks, (level 2) information extraction, (level 3) guideline-based clinical reasoning, such as TNM staging and registry coding, and (level 4) interpretive synthesis, such as explanations, summarization, or decision support. Rather than grouping studies by NLP task labels, this framework tracks how LLM applications progress from preprocessing and extraction toward higher-level interpretation and synthesis. We followed PRISMA-ScR guidelines and searched four databases through September 2, 2025, identifying 41 eligible studies. Most studies focus on level 2 tasks, with fewer addressing level 3 and level 4 tasks. Encoder-based models, including domain-specific variants such as BioBERT, were commonly used for structured extraction tasks, whereas generative models, including GPT, LLaMA, and Mistral-family models, were increasingly evaluated for prompting-based extraction, staging, and summarization. Reported performance was often high for well-defined extraction tasks, but external validation was uncommon, and metrics varied across studies, limiting direct comparison. Overall, the evidence suggests that success in lower capability levels does not consistently translate to higher-level reasoning, especially when reports are inconsistent, required staging inputs are missing, or clinical assumptions must be inferred, which helps explain gaps between benchmark results and practical adoption. Future work should prioritize robust multi-site validation, clinically meaningful error analysis, transparent evaluation, and privacy-preserving implementation strategies to support safe integration in oncology.
Maryam Seifaddini, Mohammad Beheshti, Steven Richberg et al.· Modern Pathology· 0 citations
Initial experiments on heart-failure-focused clinical question answering show that CGX improves evidence retrieval quality and perceived answer reliability over conventional retrieval methods, while reducing total graph construction time by 69.7% under the same input corpus and hardware setting.
Dat Nguyen, Anh N Le, Binh T. D. Trinh et al.· Journal of Biomedical Inform...· 0 citations