A Preliminary Study on Building an LLM-Powered Physics AI Tutor for High School Students

Abstract

Applying Large Language Models (LLMs) to Physics remains challenging due to the complex integration of domain theory, real-world modeling, and precise computation. This paper presents a pre-feasibility study evaluating LLMs as physics AI tutors, focusing on prompting strategies and computational robustness. We curated an expert-validated, anti-leakage testbed of 442 open-ended Mechanical Oscillation problems and applied a Constrained Prompt Engineering technique (Theory–Calculation–Conclusion) to systematically investigate model behavior. Empirical results, evaluated via a 4-level rubric isolating reasoning from calculation errors, reveal a critical dichotomy: while constrained models under a structured paradigm (e.g., Qwen2.5-7B-Instruct) exhibit excellent logical reasoning (96.7% reasoning capability index), they face severe arithmetic barriers. Specifically, a diagnostic stress-test on 450 mathematical variants shows accuracy plummeting from 75.3% to 33.3% when processing complex irrationals. Furthermore, maintaining logical consistency requires excessively long prompts, posing context overhead challenges. These findings establish a preliminary baseline for LLM-based tutoring, suggesting Supervised Fine-Tuning (SFT) to internalize physical logic and Tool-use integration to resolve mathematical blind spots.