Artificial Intelligence and the Accuracy of Disease Diagnosis

Abstract

Artificial intelligence (AI) and large language models (LLMs) have demonstrated strong diagnostic performance in narrowly defined medical tasks, but less is known about their accuracy when diagnosis is open ended and spans diverse disease categories. The present study evaluated GPT-4o as a general diagnostic system using 1,000 real-patient cases derived from the MIMIC-IV clinical database. Clinical information from each case was parsed and assembled into a structured summary, and GPT-4o generated a ranked list of five possible diagnoses. Three prompting strategies were compared: zero-shot, few-shot, and chain-of-thought prompting. Performance was evaluated at both the disease level and the broader medical-category level using Top-1, Top-3, and Top-5 accuracy. Zero-shot prompting produced the strongest diseaselevel performance, with Top-1 accuracy of 55.7%, Top-3 accuracy of 69.5%, and Top-5 accuracy of 73.0%. Few-shot prompting yielded corresponding accuracies of 52.1%, 65.9%, and 69.6%, while chain-of-thought prompting produced substantially lower accuracies of 34.2%, 46.7%, and 50.3%. At the category level, zero-shot Top-1 accuracy was 72.3%, compared with 69.4% for few-shot and 56.2% for chain-of-thought prompting. Zero-shot performance varied considerably across disease categories, with particularly high Top-5 disease-level accuracy for musculoskeletal (98%) and endocrine/metabolic (91%) cases and lower accuracy for cardiovascular (66%) and other (63%) cases. Accuracy also appeared to stabilize after approximately 300–400 evaluation samples. These findings suggest that GPT-4o can identify the correct diagnosis within a short differential diagnosis for a substantial proportion of heterogeneous real-patient cases, but performance depends on prompting strategy and disease category. Further research should address possible information leakage from dischargesummary framing, automated diagnosis-matching limitations, unequal category sample sizes