Skip to content

Accuracy and Short-Term Consistency of Generative AI Chatbots in Guideline-Based Endodontic Decision Support: A Multi-Model Benchmarking Study

Sep 2026 · Dental journal · Vol 14 · 0 citations · 41 references
Medicine

TL;DR

These findings support guideline-based evaluation and clinician verification when generative AI chatbots are used to provide endodontic information relevant to decision support or dental education.

Abstract

Background/Objectives: Large language model (LLM) chatbots are increasingly used for dental information and decision support, yet their accuracy and short-term reproducibility in endodontics remain insufficiently established. This study compared five chatbots using open-ended questions derived from established AAE and ESE endodontic guidelines. Methods: Twenty-six guideline-based questions were content-validated by five endodontists using Lawshe’s Content Validity Ratio. ChatGPT-4o, Gemini 2.5 Pro, DeepSeek-V3-0324, ScholarGPT (academic version built on OpenAI’s GPT-4 architecture), and MedGebra GPT-4 answered each question across three days and three sessions per day, yielding 1170 responses. Two blinded endodontists scored responses on a 5-point guideline-concordance scale. Brunner–Langer LD-F2 analyses assessed model and temporal effects, while weighted kappa evaluated response consistency. Results: The overall model effect was significant (p < 0.001). ChatGPT-4o, Gemini 2.5 Pro, DeepSeek-V3-0324, and ScholarGPT showed comparable performance, whereas MedGebra GPT-4 performed significantly lower after Bonferroni correction. The day effect was not significant (p = 0.054), while the session effect (p = 0.048) and model × session interaction (p = 0.030) were significant. Weighted kappa values varied across models and assessment days, ranging from 0.689–0.730 for Gemini 2.5 Pro, 0.606–0.662 for ChatGPT-4o, 0.520–0.645 for DeepSeek-V3-0324, 0.458–0.592 for ScholarGPT, and 0.240–0.739 for MedGebra GPT-4. Conclusions: Guideline-aligned performance and short-term reproducibility differed across the evaluated chatbots, showing that accuracy and consistency represent distinct aspects of performance. Repeated assessment captured variation missed by single-session testing. These findings support guideline-based evaluation and clinician verification when generative AI chatbots are used to provide endodontic information relevant to decision support or dental education.

Read PDF

Similar papers

Sep 2026

Guideline-Based Evaluation of Five Generative Artificial Intelligence Chatbot Platforms for Clinician-Oriented Questions on Open Temporomandibular Joint Surgery.

The findings describe informational quality, not clinical safety or decision-making, and support specialist verification before clinical use, and support specialist verification before clinical use.

Selin Gaş, Erdinç Sulukan, Büşra Korkmaz · 0 citations
Open access Sep 2026

Evaluation of large language model responses to different types of questions based on European Society of Endodontology position statements

Objectives: This study aimed to compare the performance of DeepSeek-V3, ChatGPT 4.0, ChatGPT 4.5, Gemini Advanced 2.0, and Grok in answering single-correct answer multiple-choice (SCQ), multiple-correct answer multiple-choice (MCQ), binary (true/false) (BQ), and open-ended questions (OEQ) based on the European Society...

Merve Defişet, Merve Yeniçeri Özata, Ayşenur Çam · 0 citations
Open access Aug 2026

Comparative evaluation of AI-based chatbots in answering prosthodontic questions from the Turkish Dental Specialty Exam (DUS)

Aims: This study compared the performance of the ChatGPT, Gemini, Copilot, and DeepSeek large language model–based chatbots in answering prosthodontic questions from the Turkish Dental Specialty Examination (DUS). Methods: A total of 126 prosthodontic questions from DUS examinations conducted between 2012 and 2021 were...

Bülent Kadir Tartuk · 0 citations
Open access Sep 2026

Evaluation of large language model responses to expert questions in anterior implant dentistry: quality, accuracy, and readability

AI chatbots can generate information with potential clinical relevance in anterior implant dentistry; however, variability in informational reliability persists and expert supervision remains essential before integrating such tools into clinical education.

Dalndushe Abdulai, Raghıb Suradı, Mehran Moghbel · 0 citations
Aug 2026

Evaluating Large Language Models in Endodontic Irrigants: A Structured Assessment of Accuracy, Readability, Hallucination Subtypes and Clinical Risk.

BACKGROUND Large language models (LLMs) are increasingly used by clinicians and learners for endodontic information, yet their reliability for irrigation-related knowledge remains unclear. This study evaluated the accuracy, readability, hallucination profile, clinical risk, and temporal stability of 4 general-purpose L...

Damla Erkal, Yunus Emre Çakmak, Kürşat Er · 0 citations

Related blog posts

Google DeepMind Blog Nov 18, 2026

Genie 4 is now available

We are rolling out a new Project Genie experience to eligible Google AI Ultra subscribers, opening access to our new Genie 4 world model.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.