Performance of AI Chatbots in Adult Reconstruction Education: A Comparative Evaluation Using AAOS ResStudy Questions.
Abstract
Background
Artificial intelligence (AI) chatbots are increasingly incorporated into orthopedic education, yet their ability to handle complex, clinically oriented questions remains unclear. This study evaluates the accuracy, response time, and citation reliability of contemporary AI chatbots when answering adult reconstruction questions from the AAOS ResStudy question bank.
Methods
Fifty-five multiple-choice adult reconstruction questions were extracted and categorized by anatomic region (hip vs. knee), concept (clinical presentation, prognosis, management), and AAOS-defined difficulty. Seven chatbots (ChatGPT-3.5, ChatGPT-4, ChatGPT-4o, Gemini, Gemini Advanced, Claude, and Co-Pilot) were assessed. Accuracy, response time, and citation quality (existence and relevance) were recorded. Statistical analyses included chi-square testing, t-tests, Pearson correlation, one-way ANOVA, and Kruskal-Wallis testing (p<0.05).
Results
Overall chatbot accuracy was 67.5%. GPT-4o performed best (80%), outperforming AAOS members (75%), while Co-Pilot ranked second (70.9%). Accuracy varied by concept, with highest performance in Clinical Presentation (78.6%) and lowest in Prognosis (63.1%). Difficulty strongly influenced performance, with accuracy dropping from 84.7% on very easy questions to 45.4% on difficult ones. Average response time was 3.6 seconds, and no significant correlation existed between speed and accuracy. Among citation-producing chatbots, Co-Pilot achieved 100% citation existence and relevance.
Conclusion
Modern AI chatbots demonstrate moderate overall accuracy on adult reconstruction questions, with GPT-4o exhibiting the strongest performance. While chatbots excel at descriptive orthopedic knowledge, they continue to struggle with applied reasoning tasks such as prognosis and management. These findings support their use as supplementary, rather than primary, learning tools. Improving citation reliability and reasoning capabilities may enhance future educational applications. LEVEL OF EVIDENCE: IV, Cross-Sectional Comparative Analysis.
