When Context Sensitivity Isn’t Enough: An Argument Against Likelihood-Based Probes for Evaluating LLM-Simulated Students

Abstract

Evaluating language models as student simulators currently focuses on whether their dialogue looks realistic, yet it is unclear whether these models actually respond to the quality of teaching the way human students do. However, this has not deterred prior works from using a model’s uncertainty, measured as the log-likelihood of a fixed correct answer, as a proxy for student behaviour. This approach has been used to simulate student responses, estimate question difficulty, and study “learn-ing”. In this work, we test whether this proxy can detect sensitivity to instructional quality. We evaluate five language models on expert-rated tutoring dialogues under tightly controlled conditions. All models reliably distinguish relevant tutoring context from irrelevant context. However, this contextual sensitivity does not extend to pedagogical sensitivity: higher-quality instruction almost never increases the likelihood of the correct answer, and reasoning-oriented models some-times show the opposite pattern. We argue that this failure is structural, and that exact token-level likelihood is too strict a measure as many different reasoning paths can be correct as opposed to just one fixed correct answer. Our results therefore question the use of target-answer likelihood as a validation metric for LLM-based student simulators and advocate for more robust metrics for learning proxies. 1 .