Lightweight Llama Models with Experts’ Curated RAG for Electrical Engineering Education: An Exploratory Comparison
TL;DR
The results show that the proposed RAG agent substantially improves the lightweight base model and produces transparent, syllabus-grounded answers that experts rated as correct and concise, while GPT-4.5 retains an edge on longer, multistep, and topology-intensive tasks.
Abstract
This study investigates the benefits of expanding small and open Large Language Models (Llama 3.x family) with retrieval and explicit referencing (retrieval-augmented generation), configured to refuse to respond when unsure about the answer. Additionally, we compare this augmented system with a cutting-edge commercial model (OpenAI GPT-4.5) in electrical engineering (EE) education questions. We build on an agentic RAG pipeline with retrieval from our team’s human-curated pedagogical reference document and a source-preserving, sectioned prompt that enforces citations. Six very experienced EE professors blindly assessed the answers to three exam-style questions of different types, generated by (i) the baseline model Llama 3.1 8B, (ii) our proposed RAG agent, and (iii) GPT-4.5, according to five dimensions: Correctness, Structure, Completeness, Precision, and Conciseness. The results show that the proposed RAG agent substantially improves the lightweight base model and produces transparent, syllabus-grounded answers that experts rated as correct and concise, while GPT-4.5 retains an edge on longer, multistep, and topology-intensive tasks. The local RAG agentic AI system achieved competitive performance relative to GPT-4.5 on dataset-scoped conceptual, procedural, and direct numerical electrical engineering circuit analysis educational tasks, which support the potential of curriculum-grounded open-weight models for constructing pedagogical applications.