ALRA: Adaptive Local Relational Alignment for Logit-Based Pre-training Distillation of Autoregressive Language Models
This work proposes Adaptive Local Relational Alignment (ALRA), a position-specific framework combining student proposals with teacher guidance, which replaces the teacher-mass coefficients of the two conditional terms with unit coefficients, preventing either term from being downweighted solely because its region has l...
Quang Hoang Trung, Quang Huu Hieu, Phuc-Thanh Nguyen et al.
· 0 citations