Don't Train From Scratch: Persistent Gains from Pruning-Initialised Distillation for Edge Language Models

Abstract

Preprint notice: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible. Deploying a language model on an embedded accelerator can fix its architecture, vocabulary and weight footprint before accuracy is considered. Holding the teacher, corpus, token budget and evaluation protocol fixed, we compare two initialisations of a 369M-parameter student distilled from a 4.2B-parameter teacher—(A) extraction, width-pruning the teacher's 752M sibling into the hardware-dictated shape, and (C) random initialisation of the same architecture—alongside an off-the-shelf 362M reference (B). Extraction wins at every measured budget: arm (C) never reaches, within 1.3B tokens, the perplexity that arm (A) attains after 100M tokens—a measured token-efficiency bound above 13×—and a saturating power-law fit over the measured range places the perplexity-ratio floor at 1.428. The ordering survives a corrected-objective replication and learning-rate sensitivity checks. The off-the-shelf reference retains higher general-benchmark scores but smaller adaptation gains on a ground-truth-verified corpus of structured scene questions: +0.486 vs. +0.142 accuracy under a matched protocol (at matched general-capability cost), and +0.486 vs. +0.339 at each arm's best rate, where the reference pays five times the general cost and most of its peak is answer-prior memorisation. A scene ablation credits the extracted student with +0.183 of scene-dependent accuracy, exceeding the teacher's +0.114 and the reference's best +0.125. Weight-only 4-bit quantisation packs the specialist's weights into 327 MiB—24× below the bf16 teacher—with domain accuracy unchanged within noise. Two negative results are reported: general-benchmark superiority was not achieved at this budget, and general-purpose distillation transferred the capacity to acquire domain capability rather than the capability itself. Reproducibility package (configurations, question banks, per-item logs, analysis code): 10.5281/zenodo.22256719.