Bilingual semantic correspondence through knowledge distillation and encoder combination

Abstract

The challenge of cross-lingual semantic similarity detection is a significant problem in the context of multilingual educational software tools. This paper proposes a novel approach using ensemble learning and knowledge distillation for the development of an efficient and interpretable cross-lingual semantic similarity detection model for the English-French language pair. The methodology is based on the fusion of knowledge from the MiniLM encoder representation using a lightweight attention mechanism, LaBSE encoder with support for language-independent semantic representations, and the BERT encoder with the ability to produce dense contextual vector representations. The knowledge is then distilled using a Multi-Layer Perceptron (MLP) architecture for the development of the semantic similarity detection model. The experimental results show that the ensemble architecture attains a validation F1-score of 0.930, while the knowledge distillation student model retains a robust F1-score of 0.918 with a low computational footprint (3.17M parameters, 42MB memory, 5.77ms inference). This demonstrates the viability of knowledge distillation for the transfer of ensemble-level semantic knowledge into a compact architecture for the context of resource-constrained educational tools.