SolidGeo-Bench: A Pilot Study on Category-Level LLM Performance in Chinese High School Solid Geometry

Abstract

Large language models (LLMs) are increasingly used in K-12 mathematics education, yet little is known about how their performance in solid geometry — a domain requiring spatial visualisation, cross-section reasoning and multi-step proof — varies across problem types. We present SolidGeo-Bench, a pilot set of 95 Chinese high-school solid geometry problems spanning seven problem categories (01–05, 11–12) and three difficulty levels. We evaluated four LLMs — Claude 4.8 Opus, ChatGPT 5.5, Qwen 3.7 and DeepSeek V4-R1 — through their official web chat interfaces under zero-shot (ZS) and chain-of-thought (CoT) prompting, using partial-credit scoring for multi-part questions. The most salient pattern in this sample is a gap of roughly 38 percentage points between spatial categories (01 Spatial Relationship Judgement, 02 Cross-Section Analysis; mean ZS ≈ 58%) and procedural categories (03–12; mean ZS ≈ 96%). The gap appears in all four models, ranging from 26 to 47 pp per model. CoT prompting raised overall accuracy most for Qwen 3.7 (+5.0 pp) and ChatGPT 5.5 (+4.8 pp), and little for Claude 4.8 Opus (+0.9 pp) and DeepSeek V4-R1 (+0.7 pp). Among annotated errors, Step 2 (modelling / mid-stage reasoning) was the most frequent first point of failure. Because the sample is small, spatial problems were partly selected after pilot testing, and all annotation was carried out by one evaluator, these results should be read as exploratory. We outline tentative implications for tutoring design. This record contains the preprint and the evaluation records (scores, reasoning-quality ratings and failure-step annotations for 95 problems × 4 models × 2 prompt conditions), with a script that reproduces every number in the preprint. Problem texts and full model responses are not redistributed because they reproduce third-party problem material. See README.md for column definitions.