This work cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and scores each putative selection by the mapping entropy (ME), measuring the loss of discriminatory power inherent in discarding part of the network neurons.
Abstract
Overparameterized neural networks carry far more hidden units than a task nominally requires, raising the question of which neurons are essential and whether that distinction is legible in the representation itself, without labels or gradients. We cast neuron selection as the problem of coarse-graining the hidden layer by retaining a subset of its neurons, and score each putative selection by the mapping entropy (ME). This quantity measures the loss of discriminatory power inherent in discarding part of the network neurons, and the selection that minimises the ME is taken as particularly informative. This criterion is fully unsupervised, in that it depends only on hidden-activation statistics. In teacher-student networks, ME optimisation recovers the minimal teacher-consistent representation and retains extra units in proportion to the hidden layer's residual variability; in a non-linear Gaussian process task, it selects coherent functional-class mappings whose preferred class shifts across training. On this task and on translation-augmented MNIST, ME-selected subnetworks outperform random subsets of equal size, most clearly under strong compression - linking configurational distinguishability to predictive performance.
It is shown that layer-wise generative learning can spontaneously uncover and progressively amplify class-related structure in unlabeled data and improve average clustering can coexist with reduced accessibility for a few difficult class pairs.
Patrick Krauss, Achim Schilling, Andreas K. Maier et al.· 0 citations
Machine learning systems increasingly face the need to remove the influence of entire data domains, such as toxic language, harmful behavior, or topical content, rather than isolated records. Recent work formalizes this problem as \emph{distributional unlearning}: selecting a subset of a forget domain whose removal mov...
P. Mohanty, Hao-Ran Tang, Maggie Makar et al.· 0 citations
A family of difference-informed pruning methods built upon this principle are introduced, suggesting that preserving output differences is a broadly useful and composable signal for post-training LLM sparsification.
Linghao Kong, Inimai Subramanian, Micah Adler et al.· 0 citations
A simple and broadly applicable residual guided procedure that greedily constructs the hidden layer using a closed form residual decrease criterion and yields a progressive training process with a guaranteed monotonic decrease of the training objective.
A boundary-first inference schedule that partitions a model into chunks, first coordinates hidden states at chunk boundaries, and then refines representations within each chunk is proposed, which instantiate in predictive coding networks (PCNs), a local-learning framework in which hidden activities and prediction error...
It is argued that functional parameter identifiability may provide an organizing principle for models that systematically reuse and recombine learned functions while adapting only the components that need to change.
G. Chindemi, Benjamin Grewe· arXiv.org· 0 citations
Related blog posts
MIT News · Artificial Intelligence· news.mit.eduNov 17, 2026
By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.
Microsoft Research Blog· microsoft.comNov 16, 2026
AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.
MIT News · Artificial Intelligence· news.mit.eduNov 13, 2026