Recursive Offline Distillation via Entropy Gain Sampling and Answer Mixing
Distillation of reasoning chains from Large Language Models (LLMs) into their smaller siblings is a standard training approach. Frequently, researchers do not have full access to a teacher model and its internal representations, which limits the resulting performance. We propose a recursive offline distillation method...
Andrey Goncharov, Daniil Vyazhev, С. С. Епанов et al.
· 0 citations