This work revisits on-policy Reverse Kullback-Leibler distillation and decomposes its objective into a teacher-fitting term and a student-entropy term, without introducing an explicit FKL branch, and proposes Adaptive Entropy Distillation (AED), which uses the teacher's entropy to dynamically calibrate token-level imitation strength.
Shizheng Li, Zhiyu Shen, Yuyin Lu et al.· 0 citations
Language models suffer from adversarial vulnerability, where imperceptible perturbations in inputs can mislead model predictions. Existing adversarial training methods enhance model robustness by augmenting training data with worst-case adversarial perturbations for each input. However, these approaches overlook the intrinsic causal relationships within data. We construct causal graphs to model the behavior of encoder-based language models under adversarial inputs, suggesting that adversarial vulnerability is often associated with the model’s overfitting to spurious correlations. Within the adversarial training framework, we formulate expressions for Total Effect (TE) and Natural Direct Effect (NDE), and propose a causal regularization objective based on TE-NDE alignment. Experiments on five benchmark datasets show that this causal adversarial training framework improves adversarial robustness across different settings. We further evaluate the model behavior in a controlled sentiment analysis setting, where the quantitative results provide additional evidence for the effectiveness of our method.
Bin Zhu, Yi Xie, Yanxia Feng et al.· Computational Linguistics· 0 citations