From Semantic Retrieval to Conversational Agent: A Web-Based RAG Architecture for Interactive System Dynamics Modeling

Abstract

Keyword-based semantic search performs poorly on complex knowledge repositories such as System Dynamics model databases, where users know the behavior they want to simulate but not the structural vocabulary needed to retrieve it. We replace the static search field with an interactive web-based conversational agent, built as a Retrieval-Augmented Generation architecture in which the system asks context-aware clarifying questions to narrow the search scope across multiple turns. The architecture was evaluated in an ablation study of 37 benchmark scenarios over a curated corpus of 63 models, comparing six retrieval strategies against an expert semantic baseline using Precision@5, Recall@5, MRR@5, nDCG@5 and Hit@5. Conversational refinement raised mean nDCG@5 from 0.1066 to 0.4422 and Hit@5 from 0.1892 to 0.5946. The improvement over the broad-intent baseline is significant on nDCG@5 and MRR@5 under Holm-corrected Wilcoxon signed-rank tests. This comparison aggregates the clarification exchange with the additional user input it elicits. A separate condition that bypasses the generative rewriting step bounds the contribution of that step. A residual gap to the expert semantic baseline (nDCG@5 = 0.5750) remains and is significant on MRR@5. Lexical BM25 applied to expert queries outperformed dense retrieval on every metric (nDCG@5 = 0.8053), showing that sparse matching retains a decisive advantage where the structural vocabulary is exact. We conclude that conversational elicitation is an effective mechanism for narrowing the expertise gap in this domain, and that the lexical results motivate pairing it with hybrid sparse–dense retrieval.