Skip to content
Conference Open access

When Longer Rationales Fail To Help A Small Model Solve Hard Problems

Sep 2026 · 2026 23rd International Computer Conference on Wavelet Active Media Technology and Information Processing (ICCWAMTIP) · pp. 1-4 · 0 citations · 15 references

Abstract

Chain-of-thought distillation can improve the reasoning ability of small language models, yet it remains unclear whether harder problems should receive longer rationales. We investigate this assumption using 5,383 paired MATH training problems, each containing both a short and a long rationale. A difficulty-aware policy assigns long rationales to Level 4–5 problems, while a matched-random policy controls for problem identity, mathematical subject, number of long rationales, and clipped training-token budget. We fine-tune Qwen2.5-0.5B-Instruct with QLoRA and evaluate it on MATH-500 across three training seeds. Under a 1,024-token generation budget, difficulty-aware allocation achieves 20.1% average accuracy, compared with 21.5% for matched-random allocation. Extending truncated responses to 2,048 tokens produces similar results of 20.4% and 21.8%, respectively. The difference varies across seeds, with no statistically significant advantage for difficulty-aware allocation. A manual audit of 100 problems and 600 responses confirms that scoring errors do not change this conclusion. These results suggest that dataset-defined difficulty alone is insufficient for allocating longer rationales to the tested 0.5B model.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.