Developing a multimodal library intelligent Q&A chatbot using retrieval-augmented generation (RAG) technology

Abstract

This study aims to enhance the user experience for patrons and improve library operation efficiency through implementation of an artificial intelligence (AI)-driven Q&A chatbot. Using RAG technology, this study developed Nana, a multimodal intelligent library chatbot based on open-source pretrained language models. Following multiple rounds of iteration and debugging to optimize its local documents retrieval and summarization capabilities, Nana’s performance was compared with four other library chatbots via four key metrics. The results demonstrate that the authors’ chatbot performs exceptionally well in evaluations, effectively addressing common library reference queries and optimizing librarians’ time by promptly answering frequently asked questions. In addition, the authors’ chatbot exhibits robust performance in understanding and responding to queries in atypical scenarios. As a multimodal chatbot, it is distinguished by its unique capability to process image data and user-uploaded documents, significantly enhancing its utility and user-friendliness. This research offers a practical roadmap for libraries seeking to upgrade their intelligent services. The RAG-based multimodal chatbot can efficiently handle user inquiries by retrieving local resources accurately. By reducing the burden of routine Q&A on library staff and improving service accessibility, it serves as a scalable framework for digital libraries (DL) aiming to modernize the user experience. This study advances smart digital libraries by developing a multimodal RAG-based Q&A chatbot, thereby addressing functional gaps in existing systems. Unlike conventional text-only chatbots, it features domain-adapted RAG optimization to boost the accuracy of retrieving library-specific local resources, plus integrated multimodal interaction (text, document, visual) for diverse user needs. Validated against four benchmarks, this study offers both a high-performance conversational solution and a replicable framework for domain-specific multimodal RAG chatbots in information services.