Alignment-Free Distillation for Non-Autoregressive Translation in Sparse Language Pairs

Abstract

The rapid advancement of neural machine translation has historically relied heavily on autoregressive models, which generate target sequences token by token. While autoregressive models achieve remarkable translation quality, their sequential decoding nature inherently limits inference speed. Non-autoregressive translation models have emerged as a promising alternative, enabling parallel token generation and significantly reducing decoding latency. However, non-autoregressive models often suffer from performance degradation, largely due to the multimodality problem where the model struggles to capture dependencies among target tokens. Sequence-level knowledge distillation is typically employed to alleviate this issue by training the non-autoregressive student model on simplified target sequences generated by an autoregressive teacher. Conventional knowledge distillation methods rely on strict word-level or structural alignments between the source and target representations. In sparse language pairs characterized by limited bilingual training data, acquiring robust alignments is profoundly difficult, leading to suboptimal distillation and poor translation quality. This paper introduces a comprehensive alignment-free distillation framework designed specifically for non-autoregressive translation in low-resource and sparse language pair settings. By leveraging optimal transport mechanisms and order-agnostic loss formulations, the proposed framework bypasses the need for explicit token-level alignment. Extensive empirical evaluations across multiple sparse translation benchmarks demonstrate that the alignment-free distillation approach not only matches but significantly surpasses traditional alignment-dependent distillation techniques in both translation fidelity and structural coherence.