General-Purpose vs. Domain-Specific Large Language Models in Antibiotic Clinical Decision-Making: A Double-Blind Evaluation with a 2X2 Factorial Design

Abstract

Background: Antimicrobial resistance poses a major threat to global public health. Large language models (LLMs) offer new possibilities for optimizing antibiotic prescribing decisions, but the capabilities of general-purpose versus domain-specific medical LLMs under different prompting strategies remain to be clarified. Methods: This double-blind, randomized-sequence evaluation used a 2X2 factorial design comparing four AI conditions-the domain-specific model MedGo and the general-purpose model DeepSeek V3.5, each under standard direct prompting and chain-of-thought (CoT) prompting-alongside real physician prescriptions across 59 complex inpatient infection cases. Five parallel regimens were generated per case and independently evaluated by three senior clinicians (1-5 comprehensive score and five domain sub-scores). ChatGPT 5.2 was additionally assessed as an automated evaluation tool. Results: Score ranking: real physicians > MedGo-CoT > DeepSeek-CoT > MedGo> DeepSeek (Friedman test, p<0.001). In base mode, MedGo significantly outperformed DeepSeek (Holm-adjusted p=0.040). CoT improved both models (Holm-adjusted p<0.001 for DeepSeek; p=0.024 for MedGo) and reduced score dispersion. MedGo-CoT significantly outperformed DeepSeek-CoT in individualized adjustment (adjusted p<0.001) and dosing precision (adjusted p=0.005). ChatGPT-expert correlation was negligible (overall Kendall {tau}=0.153, p=0.003; subgroup {tau}=0.06-0.20, all p>0.05). Conclusions: Domain-specific medical LLMs enhanced by CoT approach the antibiotic decision-making level of real physicians, with advantages in individualization and dosing precision. However, notable deficiencies persist in antimicrobial stewardship ecological awareness and automated evaluation reliability, underscoring the continued indispensability of senior clinical expertise.