Skip to content

Understanding and Exploiting Anisotropy in Post-Training

Sep 2026 · 0 citations · 26 references
Computer Science

TL;DR

LLM post-training combines supervised fine-tuning (SFT), a mode-covering forward-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective to exploit anisotropy, a division of labor that post-training can exploit.

Abstract

LLM post-training combines supervised fine-tuning (SFT), a mode-covering forward-KL objective, with reinforcement learning (RL), a mode-seeking reverse-KL objective. Frequency-weighted likelihood training leaves a well-known signature: \emph{anisotropy}, in which a few residual channels carry disproportionately large activations. Anisotropy is widely documented and usually treated as a defect, yet its function and its interaction with post-training remain unclear. We first analyze it. A label-free outlier rule isolates about 5\% of residual channels that are essential for language modeling: removing them raises perplexity from 10 to over $10^6$, versus 35 for count-matched random channels. Yet they barely distinguish correct from incorrect reasoning. SFT reshapes them, whereas RL leaves them largely intact and adapts the complementary channels. These channels therefore form the model's \emph{coherence substrate}, and reasoning adaptation happens elsewhere. We then exploit this. \textsc{SphereGate} learns one bounded gain per residual channel on a frozen backbone. Its activation-weighted gradients provably limit movement of high-energy coherence channels and leave the remaining channels free. With 0.1M trainable parameters, \textsc{SphereGate} outperforms parameter-efficient baselines by 2.0--7.3 points on MATH-500 across Qwen2.5 (0.5B--7B) and Llama-3-8B, is comparable or exceeds full-model GRPO. Anisotropy is not a defect but a division of labor that post-training can exploit.

View source

Similar papers

#machine learning Preprint Oct 2026

Understanding the Weight Averaging Mechanism in LLM Training for Post-Training Quantization

Large language models (LLMs) are typically pretrained in high precision but increasingly deployed with low-precision post-training quantization (PTQ). Recent studies have shown that using weight averaging during pretraining can improve PTQ performance compared with learning-rate decay, suggesting that it might provide...

Han Wang, Tianqi Shen, Zong-Lin Liu et al. · 0 citations

Label-free steering: Compressing test-time reinforcement learning into bias-only subspaces

A label-free bias-only TTRL, which uses majority-vote pseudolabels as rewards and optimizes only ~100K bias parameters while keeping the pretrained backbone frozen, demonstrating that substantial test-time adaptation can emerge from optimizing a tiny bias-only subspace using entirely label-free rewards.

Naveen Vakada, Ming-Yuan Li, Shao-Xiong Ji · 1 citation · ⚡1

Understanding LLM Parameter Update Sparsity through the Lens of Fisher

Recent studies have observed that parameter changes during language-model post-training can be concentrated in a small subset of coordinates. This phenomenon has been reported in reinforcement learning, on-policy distillation, and supervised fine-tuning on near-policy data. Its recurrence across different post-training...

Yu-Fan Zhang, Sagnik Mukherjee, Hao Peng · 1 citation
Conference Open access Sep 2026

Meta-Cognitive Resonance Label Correction for Instance-Dependent Noise

The Meta-Cognitive Resonance Label Correction (MecReC) framework introduces a novel meta-cognitive resonance mechanism that aggregates multiple complementary signals to comprehensively assess label reliability, and guarantees a tighter error bound for label purification.

Gao-Xia Jiang, Jie Su, Sen-Yu Hou et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Reasoning-Preserving Fine-Tuning of Post-RL LLMs with Null-Basis LoRA

Reinforcement learning (RL)-based post-training has become an effective approach for eliciting reasoning capabilities in large language models (LLMs). However, adapting post-RL models to new knowledge domains or behaviors through subsequent supervised fine-tuning (SFT) can severely overwrite these capabilities. Existin...

Wen-Zhi Fang, Nicholas Tzou, L. Valkov et al. · 0 citations
#artificial intelligence Preprint Sep 2026

Relative Generalization Invariance of LLM Pretraining

This work introduces Relative Generalization Invariance (RGI), the invariance of the validation-loss difference between any two tokens across models, and shows that RGI approximately holds across a wide range of optimizers and moderate architectural variations, suggesting that these choices induce an approximately unif...

Feng-Zhuo Zhang, Shu-Che Wang, Sheng-Gui Li et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

MIT News · Artificial Intelligence Aug 27, 2026

Looking beyond natural sequences

A new machine-learning framework aims to improve the success rate of computational protein design while moving away from results that reproduce sequences found in nature.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.