Do Calibrated-Probability Claims Hold Up in Clinical Diagnosis? A Pre-registered Evaluation of a Decision Model Against Frontier Language Models

Abstract

Background: A new class of AI "decision models" returns a probability for every option in a single pass and is marketed as calibrated, which invites use in clinical triage. Whether those probabilities hold up on clinical diagnosis has not been independently tested. Objective: To test the diagnostic accuracy, calibration, and probabilistic coherence of a decision model (Jev 1.13, TypeSafe AI) against frontier language models, with and without reasoning. Methods: In a pre-registered study (OSF, registered 26 September 2026), we compared Jev with Claude Opus 5.5 (adaptive reasoning, always on) and Claude Opus 5 (reasoning disabled) on three case sets: the 45 Semigran symptom-checker vignettes, 1,000 stratified synthetic patients from DDXPlus with reference differentials, and 80 simulated ophthalmic complaints (40 diagnoses at two levels of detail). Each model returned a probability for every diagnosis on a fixed list. Outcomes were top-1 and top-3 accuracy, expected calibration error (ECE), Brier score, the rate of exactly zero probability on the true diagnosis, sensitivity to option order, behavior on uninformative inputs, and behavior when the true diagnosis was absent. Results: On DDXPlus, top-1 accuracy was 72.0% for Jev, 79.5% for Opus 5 without reasoning, and 87.4% for Opus 5.5 (both comparisons with Jev P<.001). Jev assigned exactly zero probability to the true diagnosis in 62 of 1,000 cases (6.2%); neither comparator did so in any case. Calibration error did not differ between Jev and the registered comparator (ECE difference 0.007, 95% CI -0.026 to 0.038). Offered an explicit "none of these" option, Jev declined to diagnose all 30 uninformative inputs; without that option, it assigned a mean 58% to a single diagnosis. Order sensitivity was small in all models. Given the option list with no patient information, Jev placed up to 96% on a single diagnosis, while the comparators spread probability almost evenly. On the ophthalmic complaints, accuracy did not differ (Jev 88.8%, Opus 5 91.2%, Opus 5.5 88.8%), and Jev was better calibrated than the registered comparator (ECE difference -0.081, 95% CI -0.124 to -0.018). Conclusions: The decision model matched frontier models on short, distinctive ophthalmic complaints but fell well behind on multi-symptom synthetic patients, including behind a comparator without reasoning, and there it excluded the correct diagnosis outright in about 1 in 16 cases. Its probabilities signaled uncertainty only when the question offered a way to abstain. Calibration claims for clinical decision models should be tested on clinical tasks, with explicit abstention options and zero-probability rates reported.