Development of an AI-driven chatbot for medication-assisted treatment standards in Scotland

Abstract

Background Scotland faces a severe public health crisis with drug-related deaths reaching 267 per million people, ranking second globally after the United States. Medication-Assisted Treatment (MAT) represents a proven intervention for heroin addiction. However, healthcare professionals struggle with accessing and interpreting current MAT standards through fragmented information systems and time-consuming manual searches across multiple websites. Despite advances in healthcare chatbots leveraging Large Language Models (LLMs), no specialized systems exist to support MAT delivery or integrate advanced technologies like Retrieval-Augmented Generation (RAG) and Knowledge Graphs for addiction treatment. Objective To develop and evaluate an AI-driven chatbot prototype that integrates LLMs, RAG, and Knowledge Graphs to enhance healthcare professionals’ access to MAT standards in Scotland, addressing current barriers in information delivery and clinical decision-making. Methods We employed a mixed-methods approach combining a survey of 39 MAT healthcare professionals (31% response rate) and systematic literature review following PRISMA guidelines. The chatbot prototype was developed using Llama2 language model, Neo4j knowledge graphs, and custom RAG implementation. Data was ethically collected from Public Health Scotland and Healthcare Improvement Scotland websites. Performance was evaluated using BLEU and ROUGE metrics, with prototype deployment via Streamlit interface. Results Survey findings revealed significant challenges with current communication methods: only 5 of 39 respondents rated existing systems as “exceptional,” while 17 rated them as “average” or below. Primary challenges included decentralized information (n=13) and time-consuming access processes (n=8). Literature review of 14 healthcare chatbot studies identified a critical gap in MAT-specific applications. The developed prototype demonstrated moderate performance with BLEU score of 36.64, ROUGE-1 score of 0.48, and ROUGE-L score of 0.42. The knowledge graph successfully integrated 227 nodes, 136 relationships, and 8 characteristics representing comprehensive MAT standards. The system successfully retrieved relevant MAT standards information in response to queries about specific MAT standards, medication protocols, and implementation guidance Conclusions To our knowledge, this study provides the first prototype of an AI-driven chatbot specifically designed for MAT professionals, demonstrating feasibility of integrating advanced AI technologies to address information access barriers in addiction treatment. While performance metrics indicate potential for enhancing MAT information delivery, further development is needed to improve semantic understanding and response naturalness. The prototype establishes a foundation for future integration with electronic health records and broader healthcare systems, with the potential to support improved treatment outcomes for individuals with heroin addiction in Scotland, subject to longitudinal clinical validation.