Learning from Teacher Continuations at Student States
By regenerating prefixes from the evolving student, OLIVE continues improving after offline distillation plateaus while better preserving the general capabilities and plasticity of the student.
Haojin Wang, Dylan Zhang, Huai-Bo Chen et al.
· 1 citation