Skip to content

SPECTRUM: Proximal Spectral Modulation for Looped Self-Distillation

Oct 2026 · 0 citations · 21 references
Computer Science

TL;DR

These findings establish correct-solution retention as a complementary objective of recursive self-improvement (RSI) and show that generation-time intervention can improve the solution repertoire retained by subsequent students.

Abstract

A model that learns from its own outputs inherits more than their correctness: it inherits which solutions it produces. We formulate Looped Self-Distillation, a self-evolution framework for code generation in which a model repeatedly generates and learns from its own raw outputs, under a fixed information budget, without ongoing external assessment or test-based selection of the generated samples. We identify a consequential separation: correctness can improve while the breadth of correct implementations contracts. We introduce SPECTRUM, which re-estimates loss-sensitive key/value geometry from a fixed reference anchor at each round and converts it into full-rank proximal spectral modulation. All generated completions train a single student, whose subsequent inference requires no intervention. After five rounds of experiments on MBPP, SPECTRUM retains 89.9% of the initial model's 64-sample correct AST richness, compared with 66.4% for Vanilla self-distillation and 65.5% for a subspace-projection control. The advantage persists at matched correct-sample counts. Without further training or recalibration, the resulting student also achieves higher matched-correct richness than Vanilla SD on HumanEval+ and APPS Intro, demonstrating transfer of the diversity benefit. These findings establish correct-solution retention as a complementary objective of recursive self-improvement (RSI) and show that generation-time intervention can improve the solution repertoire retained by subsequent students.

View source

Similar papers

Activation-Conditioned Self-Distillation

Activation-Conditioned Self-Distillation (ACSD), which extracts a steering vector by contrasting activations of self-generated trajectories that reach verified correct answers within a generation budget with those of all remaining trajectories, achieves the highest mean accuracy over four mathematical benchmarks among...

Zhe-Xi Lu, Subhajit Chaudhury, Tejaswini Pedapati et al. · 1 citation
#artificial intelligence Preprint Oct 2026

Progressive Misalignment Can Emerge from Benign Recursive Self-Distillation

This paper observes that when a safe model is recursively self-distilled on its own (safe) generations while receiving an additional non-harmful cue, it becomes increasingly misaligned across a diverse set of harmful queries; it is called Progressive Misalignment.

Karim Abdel Sadek, Federico Di Gennaro, Fanny Yang · 0 citations

iSDFT: Information-Proximal Self-Distillation for Continual Learning in LLMs

Controling how much and when teacher information is introduced improves specialisation while preserving broader capability is shown, and results show that controlling how much and when teacher information is introduced improves specialisation while preserving broader capability.

Ahmed Khaled Khamis, Xiao-Tong Ji, Hassan Jaber et al. · 1 citation
#artificial intelligence Preprint Oct 2026

VASPO: Self-Distillation with Privileged Value Estimation

On-policy self-distillation provides dense supervision for reasoning models via a privileged self-teacher. However, this supervision reflects local token preferences rather than the expected return of an unguided student continuation. Recent methods combine this signal with GRPO. They use GRPO for update directions and...

Fei Wu, Zhen-Rong Zhang, Jian-Shu Zhang et al. · 0 citations
#machine learning Preprint Oct 2026

Learning without Overwriting: A Theory of Self-Distillation and Supervised Fine-Tuning in Continual Reasoning

Modeling LLM reasoning as search over a directed acyclic graph, a unified theoretical analysis of both the dynamics of post-training---OPSD and supervised fine-tuning (SFT) in continual learning---and the impact of pre-training on subsequent performance shows that reliable continual reasoning depends on how post-traini...

Shinichi Uemura, Taiji Suzuki · 0 citations

Related blog posts

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.