Designing Reliable LLM Systems for AI-Assisted Engineering Education

Abstract

Large Language Models (LLMs) are increasingly used in AI-assisted engineering education, enterprise learning, technical knowledge support, and software-development workflows. However, static retrieval-augmented generation (RAG) pipelines retrieve external context for every query, which can increase latency, token cost, and irrelevant-context exposure, while direct generation can produce unsupported or stale answers for policy-sensitive, documentation-dependent, or multisource questions. This paper presents a production-oriented adaptive retrieval framework that treats retrieval as a conditional reliability decision rather than a default execution step. The framework combines query embeddings, similarity-based query memory, feedback-confirmed routing labels, comparison/fallback handling, and observability logs to decide whether a query should be answered directly, routed through retrieval, or escalated for evidence comparison. The system was evaluated in a production-similar controlled test bed using a 1,300-scenario validation workload and public benchmark validation using HotpotQA and RAGTruth. In the 1,300-scenario evaluation, the proposed method achieved 91.31% routing accuracy, 85.00% grounded-answer rate, 85.38% unsupported-answer detection, and 79.85% fallback correctness, while reducing mean latency compared with always-on Static RAG. Paired statistical testing showed significant improvements over LLM-only, Static RAG, and Threshold RAG baselines for routing, grounding, unsupported-answer detection, fallback behavior, and latency. The paper also discusses deployment considerations including query-memory growth, stale no-retrieval labels, knowledge-base version tracking, privacy-preserving redaction, and instructor oversight for responsible educational use.