The relative effectiveness of a self-assessment chatbot and Chat GPT for teaching neurological diseases

Abstract

Personal assistants, chatbots, and large language models (LLMs) increasingly pervade our society. Many people turn to chatbots and LLMs for medical advice and to answer medical questions. While chatbots and LLMs will inform users of their potential to make mistakes, since following bad medical advice can be harmful, they focus primarily on conveying information rather than making sure the user understands the information. This is an important issue as well, since incorrectly following good advice can also be harmful. To address this issue, we created a chatbot to teach users about neurological diseases. Our chatbot has users self-assess what they know about the subject so that when users ask it questions, it takes the user’s existing level of knowledge into account and focuses on filling knowledge gaps. We tested the effectiveness of this self-assessment chatbot against Chat GPT. 20 Participants were taught about neurological diseases and then half used Chat GPT to help them understand what they learned and half used our self-assessment chatbot. Afterwards, both groups received a post-test to see how much they learned. Those using Chat GPT scored 73.75%, on average, on the post-test, while those using the self-assessment chatbot scored a statistically higher 83.5% on average.