This work proposes a distillation approach based on ranking supervision that consistently outperforms supervised fine-tuning as well as FKL and RKL baselines in Python code generation, multilingual generation, and data-science scenarios and offers guidance for future research in model compression.
Abstract
In recent years, large language models (LLMs) have achieved remarkable advances in code generation. However, their massive parameter scales hinder deployment in resource-constrained environments. Knowledge distillation has emerged as an effective compression technique that transfers knowledge from a large teacher model to a smaller student model, thereby reducing computational cost while retaining strong generative capability. However, traditional distillation methods usually depend on forward and reverse Kullback-Leibler (KL) divergence, aligning the probability distribution over the entire vocabulary. This process makes them susceptible to long-tail noise and often leads to weaker performance than supervised fine-tuning with labeled data. To address this issue, we propose a distillation approach based on ranking supervision. At each step, the method selects the candidate tokens with the highest probabilities from the teacher's output and applies a ListNet-based loss. This loss encourages the student to learn the teacher's ranking preferences. Unlike conventional KL distillation, ranking distillation avoids exhaustive alignment of low-confidence tokens, achieving comparable training time while significantly reducing GPU memory consumption. We conduct systematic evaluations on four public benchmarks (HumanEval, MBPP, DS-1000, and MultiPL-E). Experimental results demonstrate that the proposed method consistently outperforms supervised fine-tuning as well as FKL and RKL baselines in Python code generation, multilingual generation, and data-science scenarios. Moreover, it maintains stable performance gains across different model scales, including both the Qwen2.5-Coder and DeepSeek-Coder families. Our method provides an effective solution for distilling large language models in code generation and offers guidance for future research in model compression.
Large Language Models (LLMs) have emerged as powerful assets for recommender systems. However, deploying them as generative recommenders or zero-shot rankers at web-scale remains bottlenecked by prohibitive computational overhead and grounding challenges. In this paper, we revitalize the classic, highly efficient two-tower retrieval architecture by adapting LLMs as semantic representation backbones rather than generative engines. We introduce an LLM-native two-tower framework engineered for high-throughput, large-scale retrieval. Our architecture introduces several key innovations: a shared LLM encoder for joint user-item modeling, End-Of-Sentence (EOS) token pooling for compact sequence embedding, cross-dataset transfer learning, knowledge distillation from powerful cross-encoder teachers, and latent reasoning within the user tower. Extensive evaluation across three public benchmarks demonstrates that cross-encoder architecture outperforms current state-of-the-art (SoTA) models, while the efficient two-tower student achieves SoTA-comparable retrieval performance. Furthermore, experiments on internal large-scale production systems yield substantial topline retrieval improvements along with high resilience to model staleness and superior data scaling. Our findings demonstrate that when augmented with modern representation learning, the traditional two-tower paradigm remains an exceptionally competitive and practical solution for industrial retrieval systems.
Zhe Xu, Prachi Agrawal, Kavosh Asadi et al.· 0 citations
Large embedding models improve retrieval quality, but serving large encoders online is expensive. We study whether a compact retriever can learn teacher ranking behavior from score vectors without access to teacher hidden states. The student trains on rows built from ground-truth positives and negative candidates produced by our data generation pipeline; we evaluate student-teacher hard-negative mining separately as an extension. We use a row-centered score-vector objective, a memory-efficient implementation of uniform all-pairs PairMSE loss. On a fixed eight-task evaluation panel, our distillation protocol recovers up to 50% of the base-to-teacher gap. The distilled 0.6B student is 4.7 times faster for query encoding and 9.7 times faster for document encoding than sequential online teacher fusion. External-transfer performance after distillation remains mixed, so our evidence supports compression of teacher rankings under matched retrieval protocols.
K. Dubovikov, Martin Takác, S. Lahlou· 0 citations
Small language models are often the only option for deployment under tight latency, cost, and on-premises constraints, but they are rarely trained from scratch: a compressed model is usually recovered through knowledge distillation (KD). This recovery step largely decides the final quality, yet it is expensive. We present a practitioner's study of how to make distillation training efficient, organised around two systems contributions. First, we show that offline KD (caching the teacher's top-$K$ logits once and training the student against the cache) matches online distillation at near-identical training loss while removing the teacher from memory, running about 29\% faster per iteration, and reaching up to 41\% higher throughput on a single H200 GPU. Second, we introduce a \emph{fused, chunked KL loss} that never materialises the full vocabulary-sized logit tensor, making peak memory linear in the sequence length. This removes the memory spike that otherwise caps context length and lets us train at four times the context (32{,}768 tokens) on a single GPU. A separate output-head-only toy benchmark isolates the loss kernel and confirms its memory and iteration-rate scaling from 4K to 256K tokens. Together these make large-scale healing and hundreds of ablations affordable. We also report supporting ablations on loss design and sequence packing. We release our chunked-loss implementation: https://github.com/CompactifAI/Full-Chunked-KL-Loss.
Bakbergen Ryskulov, Iker Garc'ia-Ferrero, David Montero et al.· 0 citations
Large Language Models (LLMs) have shown remarkable potential in recommendation systems but suffer from prohibitive inference latency. Existing distillation approaches typically target Small Language Models (SLMs) or Conventional Recommendation Models (CRMs), yet face a critical trade-off between computational cost and semantic reasoning capacity. To bridge this accuracy-efficiency gap, we introduce Reasoning-to-Encoder Distilla-tion (R2END), a framework that establishes a text encoder as the optimal student architecture for scalable recommendation. Unlike meth-ods that mimic token generation, R2END compresses the teacher’s reasoning into a dense vector space via a semantic alignment objec-tive, effectively capturing user-item dynamics. Extensive experiments on four datasets demonstrate that R2END not only outperforms state-of-the-art baselines but also achieves drastically reduced latency, offering a sweet spot for recommendation.
D. Han, Daeyoung Roh, A. Y. Kim et al.· Annual Meeting of the Associ...· 0 citations
The rapid advancement of large language models (LLMs) has revolutionized natural language processing, yet their enormous parameter size presents significant challenges for fine-tuning on downstream tasks. While parameter-efficient fine-tuning (PEFT) methods such as Low-Rank Adaptation (LoRA) and Quantized LoRA (QLoRA) have substantially reduced computational requirements, they often introduce knowledge misalignment and representation degradation during adaptation. To bridge this gap, we propose GLA-LoRA, a novel PEFT framework centered on Global-Local Knowledge Alignment. Our approach establishes a unified learning strategy that synergistically integrates multi-granular contrastive learning with knowledge distillation. GLA-LoRA explicitly preserves dataset-wide semantic structures through global contrastive learning and maintains sample-level feature consistency via local invariance learning, while a teacher model guides the adaptation to retain the original model's knowledge. Extensive evaluations across eight benchmarks, spanning six GLUE tasks and two intent detection datasets, demonstrate that GLA-LoRA consistently outperforms strong baselines including standard LoRA and QLoRA. Under identical parameter budgets, GLA-LoRA achieves consistent and significant improvements over standard LoRA across three base models and eight benchmark datasets. On sentence-level classification and regression tasks, our method yields average accuracy gains of 1.4%, 1.1%, and 1.2% on LLaMA2-7B, RoBERTabase, and BERTLarge, respectively. For intent detection (CLINC and HWU), it achieves even more pronounced improvements, with up to 2.3% accuracy gain on BERTLarge. These results establish that explicit global-local knowledge alignment is essential for achieving high-fidelity, parameter-efficient fine-tuning across diverse language tasks.
Pretrained vision-language models such as CLIP excel at zero-shot recognition but often fail at compositionality, particularly attribute-object and relational structures. Recent studies mitigate this issue by augmenting training with synthetic hard negatives generated by a cascade of large language models and text-to-image models, which incurs substantial pipeline overhead. We instead propose MLLMCLIP, a heterogeneous distillation framework that transfers multimodal knowledge directly from a generative Multimodal Large Language Model (MLLM) teacher into a discriminative CLIP student, bypassing synthetic data entirely. To bridge the architectural mismatch between the two paradigms, we introduce an attention-based per-layer token selection and a CKA-based distillation loss. Compared to prior CLIP-enhancement methods, MLLMCLIP achieves state-of-the-art compositional accuracy while delivering consistent gains on standard zero-shot classification and image-text retrieval, showing that feature-level distillation strengthens both compositional and general vision-language representation capability.
Jongsuk Kim, Qiyu Wu, Zhuoyuan Mao et al.· 0 citations