Jul 2026· International Conference on Digital Health· pp. 315-323· 0 citations· 26 references
Computer Science
TL;DR
An evaluation framework based on the MedMCQA dataset consisting of two complementary uncertainty settings is proposed, which introduces linguistic uncertainty cues through prompt modifications to simulate ambiguous clinical contexts and observes significant variation across models in their ability to abstain when the correct answer is unavailable.
Abstract
Large Language Models (LLMs) have recently achieved strong performance in medical question answering and clinical reasoning tasks. However, their reliability under uncertainty remains poorly understood which raises critical concerns for deployment in high-stakes clinical settings. In such environments, incorrect predictions are inherently risky, but confident incorrect predictions can be particularly harmful as they may mislead clinical decision-making. In this paper, we conduct a systematic behavioral analysis of LLMs under clinical information uncertainty. We propose an evaluation framework based on the MedMCQA dataset consisting of two complementary uncertainty settings. First, we introduce linguistic uncertainty cues through prompt modifications to simulate ambiguous clinical contexts. Second, we construct an answer removal setting, wherein the correct option is deliberately excluded mandating the model to recognize insufficient information and abstain. We analyze both model accuracy and confidence behavior using multiple calibration metrics including calibration gap, Expected Calibration Error (ECE), and Unsafe Confident Error Rate (UCER) across 500 medical questions. Our results reveal a consistent failure mode, i.e., although accuracy degrades under increasing uncertainty, model confidence remains misaligned with accuracy. This leads to a substantial increase in unsafe confident errors, indicating that model confidence remains largely insensitive to clinically meaningful information loss. Furthermore, we observe significant variation across models in their ability to abstain when the correct answer is unavailable, with some models persistently producing high confidence hallucinated answers. These findings expose critical limitations in the epistemic reliability of current LLMs and highlight the need for uncertainty aware evaluation methods prior to their deployment in clinical workflows.
MedQAbstain is introduced, a benchmark explicitly designed to evaluate medical abstention under uncertainty, and finds that state-of-the-art LLMs systematically overcommit, rarely abstaining even when the question itself is hidden.
Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini et al.· Annual Meeting of the Associ...· 2 citations
Clinical decisions, such as diagnosing conditions, prescribing medications, and recommending procedures, are rarely made with absolute certainty. Instead, they reflect probabilistic judgments shaped by evolving patient information and incomplete evidence. However, current EHR systems and knowledge graphs encode such decisions as deterministic triples, lacking a mechanism to represent the subjective confidence inherent in clinical reasoning. We present JudgEHR, a framework for clinical decision confidence estimation that leverages large language models (LLMs) to perform cohort-based collective inference over structured patient records by representing clinical events as knowledge graph triples and integrating them into LLM prompts. JudgEHR groups related clinical concepts into cohorts using LLM-driven relational inference, and then jointly evaluates the plausibility of all clinical decision triples within each cohort by considering patient visit history and background medical knowledge. We apply our method to the MIMIC-III dataset. Our statistical analysis shows that JudgEHR generates semantically consistent confidence scores, with similar concepts receiving closer values, whereas dissimilar replacements yield large confidence differences. Experiments on the MIMIC-III dataset show that incorporating the confidence into a zero-shot LLM-based pipeline improves relative AUROC by ${1 4. 6 \%}$ and AUPRC by 21.8% on the mortality prediction task.
Kexuan Xin, Guillaume Pelat, Jonathan Vitale et al.· International Conference on...· 0 citations
Medical large language models are often judged by how many clinical questions they answer correctly. That view is useful, but it misses a practical risk. A model may know the right answer and still change its response when the same case is written in a different patient voice. This paper evaluates that risk as SDoH aware narrative anchoring bias. We use NarrativeShield SDoH MedQA, a counterfactual medical question answering dataset in which each case appears in persona based narratives while the answer key remains fixed. The dataset is reshaped from wide format into case grouped persona rows. We evaluate three open source instruction tuned LLMs from the Qwen2.5 family: 1.5B, 3B, and 7B. The final experiment uses 300 clinical cases and produces 8,100 model responses across three prompting conditions. We report persona level accuracy, counterfactual consistency, correct consistency, and narrative sensitivity error. Qwen2.5 7B achieves the best accuracy at 56.33 percent and the best correct consistency at 40.33 percent. Paired McNemar exact tests show significant accuracy gains for 7B over 3B in all prompt settings. Even so, narrative sensitivity remains, with the lowest error still at 31.67 percent. These results suggest that trustworthy clinical decision support should be evaluated by both average correctness and stability across medically equivalent patient narratives.
Machine learning models deployed in clinical decision support systems almost universally produce point predictions without any accompanying measure of uncertainty. In high-stakes healthcare settings, this is not merely a technical limitation: a miscalibrated prediction can directly influence patient management decisions with real consequences for safety and outcomes. Several uncertainty quantification (UQ) methods have been proposed to address this gap, including conformal prediction, Bayesian neural networks (BNNs), and Monte Carlo (MC) dropout; however, their comparative evaluation has predominantly been conducted under idealised conditions that do not reflect clinical deployment, where patient populations, treatment practices, and data recording procedures change over time. We present a rigorous empirical framework for comparing these three UQ approaches on two clinical prediction tasks, in-hospital mortality and 30-day readmission, using 74,829 ICU admissions from the MIMIC-IV database. Methods are assessed across three dimensions: calibration quality (ECE, ACE, Brier score), robustness under temporal distribution shift, and clinical decision utility via net benefit analysis. All experiments are replicated across five independent seeds, with comparisons made using Wilcoxon signed-rank tests with Holm-Bonferroni correction. Under standard evaluation conditions, all three methods achieve similar discriminative performance (AUROC 0.836-0.844 for mortality; 0.637-0.641 for readmission). Under temporal shift, BNN calibration degrades most sharply on the readmission task (ΔECE = 0.011 ± 0.002) compared with MC Dropout (ΔECE = 0.002 ± 0.003), while AUROC paradoxically improves for all methods, demonstrating that discriminative and calibration performance can decouple under distribution shift. Conformal prediction maintains near-nominal empirical coverage on the mortality task (0.886 ± 0.002) but shows notable violations on readmission, raising practical concerns about exchangeability assumptions in deployed systems. These findings support a more demanding evaluation standard for UQ in clinical machine learning, one that moves beyond static i.i.d. benchmarks toward temporally robust, decision-aware assessment.
Isaac Tosin Adisa, Francis Mawutor Amuyao, Ezekiel Olaoluwa Joaquim· International journal of re...· 0 citations
Early outcome prediction for intensive care unit (ICU) patients using multimodal electronic health records (EHRs) is crucial for timely intervention and resource allocation. Recent methods for clinical reasoning on EHRs using large language models (LLMs), which involve textualizing structured data and concatenating it with unstructured data for reasoning. However, these methods lose information about numerical trends because they convert structured data into static text. Furthermore, LLMs often exhibit overconfidence, producing highly certain but unreliable predictions. To address these issues, we propose GuardMLLM, a dynamic fusion framework that perceives overconfidence for multi-tasking early ICU outcome prediction.Specifically, GuardMLLM learns complex temporal relationships from structured data through temporal modeling and aggregation. Simultaneously, it transforms structured data into clinical narratives containing trend information and combines this with unstructured data. Task prediction is then performed using an LLM clinical inference engine. Furthermore, we designed a multi-indicator overconfidence detector and applied temperature calibration to quantify and correct unreliable LLM confidence. An adaptive, gated dynamic fusion module balances temporal representations and LLM-derived inference features to suppress LLM overconfidence.We verify our method on two large public medical datasets, Medical Information Mart for Intensive Care and Collaborative Research Database. Experimental results show that our method improves performance on tasks such as predicting patient mortality and ICU length of stay, and effectively alleviates overconfidence in LLM.
G. Fu, Xiaolong Xu, Haolong Xiang et al.· IEEE journal of biomedical a...· 0 citations
Automated detection of errors in clinical documentation is a promising application of large language models (LLMs), yet decisions to deploy such models rest on benchmarks that evaluate each clinical note in isolation. Error-detection benchmarks are typically constructed by injecting errors into notes, such that each erroneous note has a natural counterpart. Aggregate discriminative metrics (e.g., balanced accuracy or F1) do not exploit this structure. We show that this omission is consequential. In particular, evaluating 15 diverse LLMs on 4 standardized clinical error-detection test sets across 3 languages, we find that 13 of 15 models fall below the level of random pairwise discrimination, even while achieving F1 scores that standard practice would read as moderate. We also observe that the underlying bias patterns differ across languages: the same model can default to"no error"on one language and over-flag errors on another. To diagnose where discrimination breaks down, we further introduce a procedure to score the evidence models cite in their outputs. We find that while models consistently locate error-relevant content, they fail to produce the corresponding correct verdict on the clean counterpart. Finally, we show that F1 and pairwise accuracy are driven in opposite directions by the same underlying bias, so that ranking models by F1 may systematically promote the weakest discriminators. For safety-critical clinical NLP applications, we advocate for supplementing aggregate metrics with paired evaluations in benchmark reporting. Code and analysis scripts are available at https://github.com/healthylaife/paired-clinical-eval.