Accuracy, Usefulness, and Impact Variability of ChatGPT-4 for COPD Medication Management: A Modified Delphi Study.

Abstract

Background Chronic obstructive pulmonary disease (COPD) management is complex and rapidly evolving. ChatGPT is a large language model (LLM) shown to generate treatment plans for chronic conditions, yet its accuracy, usefulness, and consistency for COPD remain poorly characterized. This study evaluated the accuracy, usefulness, and impact variability of simultaneous ChatGPT-4.0 responses to COPD medication management questions. Methods Five COPD treatment questions were simultaneously entered into three separate computers using ChatGPT-4.0 during a single session, generating 15 total responses. Three residency-trained, board-certified clinical pharmacists rated each response across three domains - accuracy, usefulness, and impact variability - using a 3-point ordinal scale (0 to 2) via a three-round modified Delphi process. Consensus was defined a priori as unanimous agreement among all three panelists. Results Of 45 response-domain ratings, consensus was achieved in 40 (88.9%). Accuracy ranged from poor (0) to good (2), usefulness from somewhat (1) to very useful (2), and impact variability from low (0) to high (2). For one question on stable COPD pharmacotherapy, all three simultaneous responses cited a retired clinical practice guideline, resulting in poor accuracy ratings. For two questions - treating exacerbations and managing a complex case - one response per question was rated higher in usefulness than the others. Conclusion Simultaneous ChatGPT-4.0 responses to identical COPD prompts differed materially in accuracy and potential clinical impact. LLM outputs should augment rather than replace clinician judgment and must be deployed with current guideline grounding and appropriate local oversight.