Development and Query Analysis of an LLM-Powered Police Chatbot with Expert Accuracy Evaluation

Abstract

Although Conversational AI significantly improves the delivery of public information, Large Language Models (LLMs) frequently face challenges in adhering strictly to Standard Operating Procedures (SOPs) and rarely leverage interaction logs for strategic analysis. To overcome these limitations, this study presents a novel police public service chatbot built on the Laravel framework, integrating the Google Gemini API with Information Retrieval (IR) methodologies and an official SOP database. The system processes user inquiries through text preprocessing, TF-IDF weighting, and Cosine Similarity to extract relevant documents, guaranteeing responses that are both contextually accurate and SOP-compliant. Furthermore, it systematically records conversation logs to analyze public needs and trends. Comprehensive evaluation via Black Box testing, IR metrics, and qualitative reviews demonstrated superior performance. The proposed system achieved an accuracy of 95.42%, outperforming rule-based (82.14%), standard TF-IDF (89.72%), and baseline Gemini (93.41%) approaches. It also received excellent satisfaction ratings from domain experts (4.83/5) and a cohort of 100 citizens (4.73/5). In conclusion, this architecture effectively combines the conversational fluency of LLMs with the strict procedural precision necessary for robust public service administration.