MoEKD: Advancing Robustness, Performance, and Behavioral Fidelity in Compressed Code Models via Mixture-of-Experts Knowledge Distillation

Abstract

Large language models for code have achieved strong performance across diverse software analytics tasks, yet their real-world adoption remains limited by high computational demands, slow inference speeds, significant energy consumption, and environmental impact. Knowledge distillation (KD) offers a practical solution by transferring knowledge from a large model (teacher) to a smaller and more efficient model (student). Despite its effectiveness, recent studies show that models distilled from a single teacher often exhibit degraded adversarial robustness, even when robustness-aware distillation techniques are employed. These observations suggest a fundamental limitation of single-teacher distillation in simultaneously transferring high-quality and robust knowledge. To overcome this limitation, we propose Mixture of Experts Knowledge Distillation (MoEKD), a KD framework that leverages a Mixture of Experts (MoE) architecture to enable more effective and robust knowledge transfer from multiple specialized experts into a compact model. MoEKD decomposes the distillation process into expert and router training, aggregation of expert knowledge through a learned routing mechanism, and distillation from the aggregated knowledge. We evaluate MoEKD on the vulnerability detection task using CodeBERT and GraphCodeBERT models. Experimental results show that MoEKD not only improves adversarial robustness by up to 35.8%, but also enhances predictive performance by up to 13%, compared to state-of-the-art KD baselines, including Compressor and AVATAR. Moreover, MoEKD-based students exhibit greater behavioral fidelity to their teachers, indicating better preservation of the teacher's predictive behavior after distillation. Deployment efficiency and environmental sustainability evaluations further show that MoEKD-based students maintain deployment costs comparable to their single-teacher counterparts, with slightly lower inference latency, energy consumption, and carbon emissions observed in most CPU and GPU settings. Our ablation analyses show that MoEKD retains its performance and robustness advantages under more aggressive student compression, while lightweight heterogeneous experts substantially reduce offline training costs and largely preserve its performance gains, but exhibit reduced adversarial robustness. The router analysis further shows that top-2 expert selection mitigates top-1 routing errors, while routing quality influences both the aggregated knowledge and the resulting student performance. Overall, these findings highlight the effectiveness of multi-expert knowledge aggregation in addressing key limitations of existing single-teacher KD approaches.