It is established that clinical AI systems require model-aware serialization middleware, multi-layer evaluation frameworks, and capacity verification before deployment, as well as a patient-safety gap where AI fails for the patients who need it most.
Abstract
We present FHIRBench, a benchmark evaluating six FHIR clinical data serialization strategies across four frontier LLMs (Claude Sonnet 4.5, GPT-5.4, DeepSeek V3.2, Qwen3 32B) on three clinical tasks using 100 stratified synthetic FHIR R4 patient bundles. We employ two evaluation layers: token-level F1 and LLM-as-judge rubric on four clinical dimensions, yielding 7,200 evaluations per layer. Our findings reveal four results. First, serialization significantly impacts quality but the direction diverges between layers: Condensed outperforms Raw JSON on F1 for 3/4 models (Wilcoxon p < 10^-17), while Raw JSON achieves higher judge scores for 3/4 models (p < 10^-7). Narrative achieves 95% of Raw JSON's quality at 83% fewer tokens. Second, model rankings completely reverse between layers -- Claude ranks last on F1 but first on clinical quality (p = 1.0 x 10^-6), demonstrating that single-metric evaluation produces misleading model selection. Third, a significant Model x Serializer interaction (Friedman p = 0.0009) precludes universal format recommendations, with GPT-5.4 favoring Raw JSON while open-weight models favor compressed formats. Fourth, Llama 3.1 70B exhibits 100% inference failure on complex patients despite operating within its nominal context window, revealing a patient-safety gap where AI fails for the patients who need it most. These findings establish that clinical AI systems require model-aware serialization middleware, multi-layer evaluation frameworks, and capacity verification before deployment. Code and data publicly available.
The choice of FHIR-to-text serialisation format significantly impacts clinical LLM quality (Kruskal-Wallis H=163.86, p<10^-33, delta=0.24 on a 5-point scale), yet remains unstudied as a clinical deployment variable. We present FHIRBench-UK, evaluating five large language models across six serialisation formats and three clinical tasks on 100 UK Core FHIR patient bundles (18,000 scored prompts across clean and perturbed cohorts). Our findings converge with independent work on open-weight models (Pator, 2026). The optimal format is context-dependent: raw_json dominates for clinical QA, hybrid_adaptive for clinical reasoning, and structured_markdown for summarisation. In 58% of model-task-complexity scenarios, raw_json is suboptimal. Model capability moderates format sensitivity: Claude Sonnet 4.5 shows 0.10-point sensitivity versus Llama 3.3's 0.39, making adaptive serialisation most valuable for budget-constrained deployments using mid-tier models. All findings replicate under clinically realistic data perturbation. The study additionally confirms a complete ranking inversion between token-level F1 and clinical quality (rho=-0.90), replicating US findings across UK Core profiles. We recommend task-aware serialisation routing as a zero-cost quality intervention for NHS FHIR-based LLM deployments.
Large language models (LLMs) show strong generalization, yet their ability to interpret structured medical data remains insufficiently studied. This work evaluated four LLMs—Claude Sonnet 4, Gemini 2.5 Pro, GPT-4o, and LLaMA 3.1-70B—using comprehensive health checkup data from the Korean National Health Insurance Service. Multiple prompting strategies (few-shot, role-based, constraint-based, and Chain-of-Thought) were tested. Zero-shot accuracy averaged 0.69 (SD 0.06), increasing to 0.92 (0.06) with combined strategies and to 0.95 (0.07) with Chain-of-Thought. Claude Sonnet 4, Gemini 2.5 Pro, and GPT-4o achieved the highest accuracies (≥ 0.98), while LLaMA 3.1-70B showed lower but improvable performance. Item-level analysis of 10,000 cases demonstrated near-perfect accuracy (0.99–1.00) for most biochemical markers, including glucose, cholesterol, triglycerides, and liver enzymes. In contrast, blood pressure showed lower accuracy (0.61–0.91), with age-related decline, likely due to the complexity of multi-categorical thresholds requiring integration of systolic and diastolic values. Subgroup analyses revealed model-specific biases: sex-related biases were observed in body mass index (Claude Sonnet 4) and urine protein, serum creatinine, and gamma-glutamyl transferase (LLaMA 3.1-70B), while age-related biases were identified in blood pressure (Claude Sonnet 4, Gemini 2.5 Pro) and low-density lipoprotein cholesterol and hemoglobin (LLaMA 3.1-70B). Overall, advanced prompting markedly improved model performance, and top-tier LLMs demonstrated robust interpretive capability, while caution is needed for variables with complex clinical semantics such as blood pressure.
Jiwon You, Hangsik Shin· npj Digital Medicine· 0 citations
ObjectiveLarge language models (LLMs) have the potential to revolutionize how population-level health research is conducted by automatically abstracting data that would otherwise be unavailable. However, few results are available on real clinical notes. We developed an LLM leaderboard showing how open-source LLMs perform at entity extraction on unseen clinical notes.
ApproachEMR data, including free-text notes, were linked to a chart-review cohort comprising 10,659 adults admitted to a hospital in Calgary, Canada, between 2017 and 2022, with data on comorbidities. We then attempted to replicate this chart review with multiple open-source LLMs in a secure computing environment. Chart review results served as the reference standard.
ResultsThere was a wide variation in performance among the examined LLMs: the smallest, Llama 3.2 3B, had a high mean sensitivity of 0.97 but low PPV at 0.4; the largest, Llama-3-70B, showed a high mean sensitivity of 0.96 and greatly improved PPV of 0.7; in-between these in size, phi 4 demonstrated a more balanced performance with a mean sensitivity of 0.81 and PPV of 0.77. However, the results varied considerably across conditions, with quirks specific to each model.
ConclusionsLLMs are already available that can perform entity extraction well enough to be considered in place of some administrative data. With rapid developments in the field, a leaderboard based on real clinical data is vital for informing researchers on best practices for integrating the latest AI techniques into their data practices.
E. Martin, Seungwon Lee, K. Riazi et al.· International Journal of Pop...· 0 citations
Large language models achieve strong scores on medical benchmarks, yet these benchmarks evaluate each question in isolation, providing no measure of whether a system can distinguish clinically similar presentations requiring different interventions. We introduce MamaBench, the first counterfactual benchmark for maternal and paediatric AI: 434 expert-authored clinical narratives in 217 pairs across 371 pathologies, evaluated via the Bias Trap Rate (BTR), the conditional probability that a model fails the counterfactual given success on the base case. We propose Evidence-Anchored RAG (EA-RAG), a three-stage retrieval method that replaces aggregate similarity with an evidence coverage objective through clinical parameter extraction, coverage auditing, and contrastive sub-queries. Across eight configurations of four frontier LLMs, base accuracy overstates robust accuracy by 16-28 percentage points in every model. EA-RAG achieves 20.3% BTR and 65.0% robust accuracy on Claude Sonnet 4.6, a 5.5 percentage point BTR reduction without degrading base accuracy. The residual 20% BTR confirms that counterfactual robustness in clinical AI remains an open challenge. Keywords: counterfactual evaluation, clinical AI, maternal healthcare, retrieval-augmented generation, diagnostic robustness
Thanni Adewuyi, Anuoluwa Sotome, Samuel Okoko et al.· 0 citations
Claude Fable 5, Anthropic's most capable publicly available model, is evaluated across eight biomedical benchmarks, four text and four multimodal, using deterministic scoring against fixed answer keys throughout.
D. Okonkwo, Magnus D. Hodgson, T. David et al.· 0 citations
This paper provides an overview of Task 6 from the Social Media Mining for Health/Health Real-World Data shared task (#SMM4H-HeaRD 2026), which focused on predicting TNM staging from pathology reports from TCGA. Seven teams submitted systems spanning fine-tuned clinical encoders, open-source generative LLMs, and closed-source API models. On a straightforward test set, most teams achieved near-perfect F1 scores (average 0 . 993 , 0 . 972 , and 0 . 957 for T, N, and M). However, on a harder tiebreak set where explicit TNM notation was removed and staging had to be inferred from clinical descriptions, performance dropped substantially (average 0 . 725 , 0 . 783 , and 0 . 846 ). Notably, the two teams using large closed-source API models generalized best to the harder set, achieving the highest T and N scores despite not leading on the easy set. These results suggest that while fine-tuned domain-specific encoders excel at surface-level extraction, larger general-purpose LLMs may be more robust when staging must be inferred from contextual clinical findings. All teams surpassed baseline overall performance on both test sets.
J. M. Acitores Cortina, Jacob S. Berkowitz, Nadine A. Friedrich et al.· Proceedings of the 11th Soci...· 1 citation