Knowledge distillation trains a student language model to match a teacher's predictions on the training text. We ask whether learning the teacher's behavior beyond the training text can further improve the student. We investigate this using three input perturbations: additive token noise, blending two documents, and bl...
Zi-Yang Cai, Yi-He Dong, Dimitris Papailiopoulos et al.· 0 citations
This work trains 1B-parameter full-bandwidth transformers on up to 400B tokens and finds that latent feedback improves validation loss, 5-shot language-model evaluation, math and coding generation, and instruction-tuned performance.
Xi Wang, Zi-Yang Cai, Zheng Zhan et al.· 6 citations
Transformers with Temporal Middle-Layer Recurrence (T2MLR), a transformers-based latent reasoning architecture that fuses a cached middle layer representation from the previous token directly into an earlier layer of the current token position, enabling abstract intermediate computation to persist across decoding steps...