Sep 2026· Studies in Health Technology and Informatics· Vol 340, pp.
234-241
· 0 citations
Medicine
TL;DR
Five current large language models could reproduce expert-validated statistical analyses from a published clinical workflow across three statistical tasks: descriptive table generation, Kaplan-Meier survival analysis, and Cox proportional hazards modelling.
Abstract
Statistical analysis of clinical data requires expertise in medical statistics. Large language models (LLMs) are increasingly used for code generation and may support both descriptive and advanced analyses, but their reliability remains uncertain. This study evaluated whether five current LLMs (GPT 5.3, Claude Sonnet 4.6, Gemini 2.5 Flash, Perplexity, and Grok) could reproduce expert-validated statistical analyses from a published clinical workflow across three statistical tasks: descriptive table generation, Kaplan-Meier survival analysis, and Cox proportional hazards modelling. All models received identical datasets and standardised prompts. Their outputs were compared with analyses performed by two experts trained in mathematical statistics and evaluated for grouping correctness, numerical accuracy, missing-value handling, and quality of generated R code. All five models produced correct descriptive statistics once dataset variables were specified explicitly. Two models failed the initial descriptive benchmark because of variable-name ambiguity, but both recovered after prompt clarification. All models reproduced the correct Kaplan-Meier p-value, although figure completeness differed. In the Cox benchmark, four models reproduced all required hazard-ratio terms, while one omitted the interaction results. These findings suggest that LLMs may support clinical research workflows, but their outputs still require careful validation before use.
HeartVar demonstrates that an LLM constrained by a domain-specific prompt and grounded in structured evidence can produce variant interpretations of first-pass quality for a cardiovascular disease cohort; however, it is not intended to replace manual assessment by a qualified variant curator.
Jamie-Lee M. Thompson, Debjani Das, Sally L. Dunwoodie et al.· bioRxiv· 0 citations
Background Large language models (LLMs) are increasingly used to support statistical analyses in biomedical research. However, their ability to accurately and reproducibly compute descriptive statistics directly from datasets has received limited evaluation. Objective To compare the accuracy, within-modality repeatabil...
P. Sebo, Ting Wang· American Heart Journal Plus:...· 1 citation
PurposeLarge language models (LLMs) are increasingly used in clinical data workflows, but most clinicians lack the time, training, or support to build predictive models. We evaluated whether a ChatGPT-assisted workflow could enable clinicians to independently construct and execute a basic predictive model using routine...
J. Caron, O. Rodriguez, Leila Raden et al.· The American surgeon· 0 citations
BACKGROUND
National Surgical Quality Improvement Program (NSQIP) data collection depends on labor-intensive manual chart abstraction, limiting efficiency, increasing cost, and necessitating patient sampling. This study evaluated whether a large language model (LLM) could accurately abstract unstructured NSQIP breast re...
Tahera Alnaseri, Yasmine Ibrahim, Ansgar Grunseid et al.· Journal of the American Coll...· 0 citations
An LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.
B. Collaço, Nadia G. Wood, Yun-Guo Yu et al.· Bioengineering· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Senior Research Scientist Jim Glass discusses how his MIT-IBM Computing Research Lab collaboration connects perception and language, and addresses the shortcoming of current models.
MIT News · Artificial Intelligence· news.mit.eduNov 20, 2026
Enforcing physics-based constraints prevents AI models from generating unreliable predictions for complex nonlinear systems in weather, climate, and turbulent flows.
An MIT and Dana-Farber Institute-led team used large language models to examine nearly 4,000 studies and hundreds of code repositories, revealing gaps in the information needed to scrutinize and reuse published research.
We use cookies to run the site and, with your consent, for analytics and to show ads.
See our Cookie Policy.