Selective Circuit Preservation During Compression of Large Language Models for Edge Deployment
TL;DR
This dissertation asks whether compression can be made capability-aware by borrowing the unit of analysis from mechanistic interpretability, and presents the problem formulation, the gap identified in a targeted review of the compression and interpretability literatures, and the design of a first experiment measuring circuit damage under quantization.
Abstract
Running a language model on an edge device means fitting it into strict memory and latency budgets. Compression, chiefly pruning and quantization, is the standard way to achieve this, but it is capability-agnostic: it minimizes average error over the model’s overall behavior rather than the retention of any particular skill, and the aggregate metrics used to report compression quality conceal which skills were lost. The damage is amplified in exactly the small models edge deployment requires. This dissertation asks whether compression can be made capability-aware by borrowing the unit of analysis from mechanistic interpretability. The proposed pipeline localizes the circuits implementing a fixed capability profile, protects the parameters those circuits occupy while the remainder of the model is compressed, realizes that protection at a granularity the target hardware can execute, and verifies both behavioral retention and mechanistic faithfulness afterwards. I present the problem formulation, the gap identified in a targeted review of the compression and interpretability literatures, and the design of a first experiment measuring circuit damage under quantization.
