Natural Language Processing for Scalable Educational AI Systems

Abstract

Recent advances in natural language processing (NLP), and in generative large language models (LLMs) in particular, have created new opportunities to develop scalable educational AI systems that support assessment, feedback, and personalized learning. Realizing this promise, however, requires more than applying general-purpose LLMs to classroom settings. Educational AI systems must generate pedagogically useful assessments, answer student questions reliably, and operate under real-world constraints of cost and latency. This dissertation studies the scalability of educational AI systems along three dimensions: content generation, answer reliability, and efficient deployment. First, we study the scalability of educational content generation. Multiple-choice questions (MCQs) are widely used in educational settings, yet constructing effective distractors, i.e., plausible but incorrect answers, is challenging and time-consuming. We introduce DiVERT (Distractor Generation with Variational Errors Represented as Text), a variational framework that jointly learns interpretable representations of student errors and generates the distractors that follow from them, producing realistic distractors for math MCQs that anticipate real student misconceptions. Second, we study the scalability of reliable educational question answering. We introduce SyllabusQA, the first open-source dataset for course logistics question answering, consisting of 5,078 open-ended question-answer pairs derived from 63 real course syllabi across 36 majors. We benchmark several strong baselines on this task, ranging from LLM prompting to retrieval-augmented generation. To evaluate factual reliability in high-stakes educational settings, we introduce Fact-QA, a novel LLM-based evaluation metric for the factual accuracy of generated answers. Third, we study the scalability of efficient educational AI deployment. We introduce RADAR (Reasoning-Ability and Difficulty-Aware Routing), a psychometrics-inspired, interpretable routing framework that adaptively allocates inference cost based on query difficulty and model ability, enabling the deployment of capable educational AI systems at a fraction of the cost of frontier models. Together, these contributions demonstrate that scalable educational AI systems require advances across content generation, answer reliability, and efficient deployment, and contribute new methods, datasets, and evaluation tools that together advance the goal of scalable educational AI.