Supplementary Dataset and Benchmark Logs: From Semantic Retrieval to Conversational Agent

Abstract

This repository contains the supplementary materials and experimental data supporting the research article: "From Semantic Retrieval to Conversational Agent: A Web-Based RAG Architecture for Interactive System Dynamics Modeling". The dataset is divided into two primary components: the source model environment (search space) and the raw experimental benchmarks evaluating retrieval performance across different levels of user expertise and conversational search strategies. 1. Model Corpus, Queries, and Scenarios This section contains the definitions, domain classifications, and configurations used to build the semantic search environment and simulate user interactions. System Dynamics Models: Contains the extracted, curated, and serialized structural definitions of 63 System Dynamics models. These models cover diverse application domains, including Ecology, Macroeconomics, Smart Cities, Agriculture, and Epidemiology. User Queries and Intents: A dataset contrasting authentic broad novice search intents (e.g., "Show me health-related models") with theoretically perfect, expert-formulated structured queries requiring specific domain vocabulary. Benchmark Scenarios: 37 standardized benchmark scenarios engineered to evaluate cross-disciplinary semantic and lexical search performance across the system. Relevance judgments were established a priori by two domain experts, independently of any system output, and comprise 95 scenario–model relevance pairs 2. Experimental Benchmarks The benchmark execution logs provide a quantitative comparative analysis of different retrieval paradigms, running on a local AI ecosystem with direct CPU inference. File: conversational_rag_benchmark_metrics.csv: This file contains the aggregate Information Retrieval metrics (Precision@5, Recall@5, MRR, and nDCG@5) calculated for the 37 test scenarios. Note that the MRR_Mean column is computed over the full retrieval list (L = 10), whereas the paper reports MRR at the evaluation cutoff k = 5; two rows are affected (scenario 18, Method C, 1/7; scenario 37, Method E, 1/6), where the first relevant document falls beyond the top five, and setting both to zero reproduces the Table 5 values exactly. File: inference_latency_logs.csv: Documents the execution timestamps and hardware latency logs for the local ONNX inference engine, tracking the multi-turn conversational delays. File: ablation_study_p_values.csv: Contains the statistical hypothesis testing (paired t-tests) results validating the significance of the agentic retrieval improvements. File: contextless_retrieval_test.csv: Contains the isolated experimental data evaluating the impact of conversational memory (Method F). Evaluated Methodologies (Ablation Study) The benchmark data tests the following six retrieval paths: Method A: Broad Intent (Direct Retrieval Baseline) using standard single-turn semantic search. Method B: Agentic Refinement (Real Multi-Turn Agent Path) representing the complete conversational architecture. Method C: Expert Semantic Baseline (Direct Retrieval), establishing semantic search performance under optimal input conditions. Method D: Apache BM25 (Lexical over Expert Query) testing exact keyword matching. Method E: Expert Query via Agent (Single Agent Turn) to assess system robustness against over-complication. Method F: Contextless User Refinement (Direct Retrieval), submitting the user's raw Turn 2 answer directly to the vector database, thereby bypassing both the conversational history and the generative query rewriting step. Key Finding - Retrieval Accuracy: Replacing the static search baseline (Method A) with the Agentic Orchestrator (Method B) improves mean nDCG@5 from 0.1066 to 0.4422, a rise of over 300%. Expressed as retrieval success, Hit@5 rises from 0.1892 to 0.5946. Key Finding - Lexical vs. Semantic Dynamics: Under optimal conditions with expert queries, exact lexical matching (Method D) outperforms dense retrieval on every reported metric, achieving an MRR@5 of 0.8784 and an nDCG@5 of 0.8053 against 0.6856 and 0.5750 for semantic search (Method C). Key Finding - Computational Latency: The logs document the latency overhead of local CPU processing. A complete multi-turn exploratory session (Method B) averages 59.33 s (SD = 23.02 s), whereas structurally complete expert queries (Method E) execute in 49.81 s (SD = 9.84 s). Each scenario was executed as an independent cold-start process, so these values include ONNX session initialisation and constitute an empirical upper bound rather than steady-state deployment latency.