Evaluating the Consistency and Reliability of an AI Diagnostic Tool in Dermatology: A Comparison Between Patient- and Dermatologist-Presented Cases

Abstract

Introduction: Large language models (LLMs) such as ChatGPT-4.1 are increasingly used for dermatologic self-guidance and clinical decision support. Objectives: To evaluate how patient- versus dermatologist-generated clinical descriptions, with and without photographs, influence the diagnostic performance and treatment recommendations of ChatGPT-4.1. Methods: This prospective observational study included 412 consecutive patients presenting to a tertiary dermatology clinic. For each case, ChatGPT-4.1 was queried under four conditions: (1) patient description alone, (2) patient description + clinical photograph, (3) dermatologist description alone, and (4) dermatologist description + clinical photograph. Outcomes included diagnostic accuracy, referral recommendations, follow-up question prompting, and appropriateness of treatment suggestions. Results: Diagnostic accuracy differed significantly across input formats (p < 0.001): patient description (53.2%), patient + photo (61.9%), dermatologist description (72.6%), and dermatologist + photo (72.3%). Correct treatment recommendations increased from 23.8% (patient description) to 48.8% (dermatologist + photo) (p < 0.001). Patient-generated inputs prompted markedly more referral recommendations and follow-up questioning (p < 0.001). Conclusion: Dermatologist-generated descriptions produce substantially higher diagnostic and treatment accuracy than patient descriptions. Current LLMs should be used as clinician-guided decision-support tools rather than autonomous diagnostic systems.