How Successful Is Artificial Intelligence in Hand Surgery Questions of the Turkish Orthopedics and Traumatology Board Exams?
TL;DR
Current LLMs demonstrate a high level of factual knowledge in hand surgery board examination questions, and while these models cannot replace formal medical training or clinical judgment, they may serve as supportive tools in orthopedic education and exam preparation.
Abstract
Aim: Artificial intelligence (AI) applications are increasingly used in medical education and clinical research. The purpose of this study was to examine the accuracy of various large language models (LLMs) in responding to questions related to hand surgery. This evaluation was based on items derived from the Turkish Orthopedics and Traumatology Board Examination administered between 2010 and 2025.Methods: A total of 220 hand surgery–related multiple-choice questions were extracted from national board examinations administered between 2010 and 2025. Questions were posed to three LLMs (ChatGPT-5.0, Gemini-Pro, and DeepSeek-V3) using both collective and individual questioning approaches across three separate sessions. Model responses were compared with the official correct answers, and success rates were calculated descriptively.Results: All three LLMs achieved satisfactory performance across all years, with success rates ranging from 73.6% to 90.9%. Individually asked questions yielded higher average scores compared with collectively asked questions. Year-by-year analysis demonstrated that all models met or exceeded the examination passing threshold throughout the 16-year period.Conclusion: Current LLMs demonstrate a high level of factual knowledge in hand surgery board examination questions. While these models cannot replace formal medical training or clinical judgment, they may serve as supportive tools in orthopedic education and exam preparation.