A side effect that misrepresents patients is measured: a one-sentence DEI prompt appended to a medical question leads models to add patient demographic attributes the question never stated, in effect rewriting who the patient is.
Abstract
Clinical-AI guidance increasingly recommends prompting language models to reason with attention to diversity, equity, and inclusion (DEI). We measure a side effect that misrepresents patients: a one-sentence DEI prompt appended to a medical question leads models to add patient demographic attributes (race, socioeconomic status, sex) the question never stated, in effect rewriting who the patient is. We call this demographic injection. Across 47 models, four medical benchmarks, and 376,000 responses scored by a validated model-judge pipeline, a single DEI prompt raises the injection rate from 0.7% to 33.1% (47x) in all 47 of 47 models, attributable to the equity content rather than to added length (18x above a length-matched control; p=1.4x10^-14). Most added content is a general population statement that leaves the answer unchanged, but a smaller subset attaches an attribute to the specific patient or changes the selected option (0.25-2.4% of responses, 99.8% toward the incorrect option), where the invented demographic changes the answer the model recommends. Phrasing scales the effect from 14% to 56%. DEI prompts are just one example of a more general mechanism. Any instruction that nudges how a model reasons can make it add unrequested details, including details about the patient. Flagged outputs are treated as model errors under study, not clinical guidance.
Five widely used prompting strategies across five influential LLMs in the latest medical bias benchmark reveal substantial heterogeneity in both effectiveness and overhead across models, with no strategy proving universally effective and some even exacerbating bias.
Ying Xiao, Zhenpeng Chen, Jie M. Zhang· Philosophical transactions....· 1 citation
MedQAbstain is introduced, a benchmark explicitly designed to evaluate medical abstention under uncertainty, and finds that state-of-the-art LLMs systematically overcommit, rarely abstaining even when the question itself is hidden.
Alessio Cocchieri, Luca Ragazzi, Giuseppe Tagliavini et al.· Annual Meeting of the Associ...· 2 citations
VITA's advantages in accuracy and completeness persisted under the neutral judge; its communication scores were lower, and this results indicate that a purpose-built clinical RAG system remains competitive with frontier LLMs on an open benchmark, consistent with corpus specificity as a design variable that improves grounding at some cost to communication polish.
Praveen Reddy, C. Mandke, Suvrankar Datta et al.· 0 citations
Safety evaluation of large language models is conducted predominantly in English and predominantly on frontier systems. Neither condition describes how such models are encountered in low-resource health settings, where small quantised systems are run locally and queried in local languages. We ask whether clinical safety established in English transfers to Hausa, and whether any failure is attributable to the language, the clinical task, or the class of model that low-resource deployment admits. Matched English-Hausa question pairs were built for three conditions of high burden in northern Nigeria: malaria, sickle cell disease, and tuberculosis, probing knowledge recall, emergency triage, a leading question inviting a contraindicated action, and a traditional-remedy claim. Six models were evaluated: five locally deployable systems of 4-9 billion parameters, two medically fine-tuned, and one frontier system. All 128 responses were scored against Nigerian national treatment guidelines by two fluent Hausa speakers working independently and blind to one another. Among locally deployable models, mean clinical correctness fell from 1.57 in English to -0.03 in Hausa, on a scale where 2 denotes a correct answer and -1 an actively harmful one. The frontier model moved from 2.00 to 1.75 and produced no response judged harmful in either language. Drift was consistent across all three conditions. Inter-rater agreement was substantial for clinical correctness (kappa = 0.70); agreement on harm was initially poor (kappa = 0.22) and is examined in detail. Because a frontier model answers the same questions competently in Hausa, the deficit is a property neither of the language nor of the clinical material, but of the deployable tier.
Anthonio Oladimeji Gabriel, Dimeji AbdulSobur Olawuyi, T. Ajayi et al.· 0 citations
ObjectiveLarge language models (LLMs) have the potential to revolutionize how population-level health research is conducted by automatically abstracting data that would otherwise be unavailable. However, few results are available on real clinical notes. We developed an LLM leaderboard showing how open-source LLMs perform at entity extraction on unseen clinical notes.
ApproachEMR data, including free-text notes, were linked to a chart-review cohort comprising 10,659 adults admitted to a hospital in Calgary, Canada, between 2017 and 2022, with data on comorbidities. We then attempted to replicate this chart review with multiple open-source LLMs in a secure computing environment. Chart review results served as the reference standard.
ResultsThere was a wide variation in performance among the examined LLMs: the smallest, Llama 3.2 3B, had a high mean sensitivity of 0.97 but low PPV at 0.4; the largest, Llama-3-70B, showed a high mean sensitivity of 0.96 and greatly improved PPV of 0.7; in-between these in size, phi 4 demonstrated a more balanced performance with a mean sensitivity of 0.81 and PPV of 0.77. However, the results varied considerably across conditions, with quirks specific to each model.
ConclusionsLLMs are already available that can perform entity extraction well enough to be considered in place of some administrative data. With rapid developments in the field, a leaderboard based on real clinical data is vital for informing researchers on best practices for integrating the latest AI techniques into their data practices.
E. Martin, Seungwon Lee, K. Riazi et al.· International Journal of Pop...· 0 citations
It is found that LLM access enhances performance on standardized clinical vignettes in all three countries, and policymakers should prioritize structured integration of LLMs as decision-support tools, combined with targeted training, local validation, and safeguards against automation bias rather than relying on access alone.
N. Rounding, L. S. Arif, Janine Berg et al.· 0 citations