A Good Rubber Duck Does Not Quack: Designing Socratic Scaffolding in AI Tutors

Abstract

Large Language Models in programming education face a fundamental tension: while capable of generating correct solutions instantly, doing so undermines the struggle necessary for conceptual learning. We present Socratic AI, a VS Code-integrated tutor that addresses this through pedagogically-grounded Socratic dialogue constrained to withhold direct solutions. Our core innovation is a stateful misconception detection mechanism that tracks student understanding across dialogue turns. Operating on a taxonomy of foundational programming errors (off-by-one, mutation vs. reassignment, return vs. print, async/parallel confusion), the system evaluates each student utterance to produce per-misconception verdicts (reinforced, weakened, introduced, absent) with certainty scores. These accumulate through small, incremental confidence score updates until a derived resolution threshold is achieved. This enables the tutor to persist on specific conceptual gaps rather than abandoning them when students deflect or express frustration. This stateful tracking enables two key capabilities: (i) the system adapts questioning intensity based on detected learner confidence signals, shifting from diagnostic probes to conceptual contrasts, and (ii) enforces purely Socratic constraints through multi-layered validation that blocks code generation, direct explanations, and multi-question responses. The system is implemented as a VS Code extension with Gemini-based classification back-end and cloud database persistence for longitudinal analysis. In a preliminary deployment with 48 students, we observed increased question specificity, frequent learner articulation consistent with mental model revision, and reduced recurrence of targeted errors on similar tasks. We contribute a deployable architecture, a misconception taxonomy, and evidence that stateful tracking enables adaptive Socratic dialogue that scaffolds productive struggle rather than short-circuiting learning.