Skip to content

Structural Limits of the Information-Theoretic Uncertainty Decomposition

Sep 2026 · arXiv.org · Vol abs/2609.39591 · 0 citations · 23 references
Computer Science

TL;DR

It is discovered that significant portions of the assumed AU, EU range are infeasible in finite settings, and cannot be attained with any class probabilities, and the infeasible region's boundary helps explain epistemic collapse.

Abstract

Uncertainty estimation in machine learning typically decomposes uncertainty into aleatoric uncertainty (AU) and epistemic uncertainty (EU) using the standard information-theoretic framework. However, in practice, two critical issues arise: entanglement (AU and EU are highly correlated) and epistemic collapse (EU magnitude shrinks with increasing model capacity). We analyze this framework on a functional level and discover that significant portions of the assumed AU, EU range are infeasible in finite settings, and cannot be attained with any class probabilities. We characterize how this infeasible region scales with the number of classes and Monte Carlo samples $N$ (e.g., from ensembles with $N$ members), revealing it is bounded by $\text{AU} \leq \log(2)/N$. Crucially, the infeasible region's boundary helps explain epistemic collapse: when model confidence is high, $\text{AU}>\text{EU}$ is guaranteed by this fundamental structural limitation. Our findings show that increasing ensemble size mitigates epistemic collapse by reducing the infeasible area. Lastly, we caution against interpreting AU and EU as independent quantities in low AU regimes, since we show they are coupled when $\text{AU} \leq \log(2)/N$.

View source

Similar papers

Preprint Nov 2026

Specification-Aware Predictive Uncertainty Decomposition

Classical decompositions of predictive uncertainty, such as the information-theoretic one, are defined for a fixed representation $x$ of the input object. They do not consider that $x$ may be incomplete, nor do they measure how much the missing information can affect the prediction. In Large Language Models, users can...

Roman Vashurin, Mikhail Seleznyov, Assel Yermekova et al. · 0 citations
Preprint Oct 2026

Sub-Gaussian Confidence Intervals for Heavy-Tailed Data: Characterizing the Limits of Inference

We investigate the problem of uncertainty quantification in the mean estimation framework. Given an i.i.d. sample from a distribution with unknown mean $\mu$ and variance $\sigma^2$, we want to construct an estimator $\wh\mu_n$ and provide a computable and size-optimal upper bound for the error $|\wh \mu_n - \mu|$. Whi...

I. Hammouda, Stanislav Minsker, M. Ndaoud · 1 citation
#machine learning Preprint Oct 2026

Regret-Optimal Wasserstein-Robust Linear Regression

Distributionally robust optimization (DRO) minimizes worst-case risk under distributional uncertainty, but its decisions can be suboptimal in terms of their excess risk, or regret. We study this gap for multivariate least-squares regression, assuming that the observed source law and unknown target law differ by an adve...

Sloan Nietert, Daniel Kuhn · 0 citations
Preprint Oct 2026

Residual-Information Generalization Bounds Under Selective Privacy: Extended Version

This work derives complementary task-compression and individual-sample refinements, identifies a sharp obstruction showing why selective privacy alone need not give a vanishing certificate, and invert the bound to obtain the exact leakage allowance for a prescribed generalization tolerance.

A. Zamani, Parastoo Sadeghi, Mikael Skoglund · 0 citations
#artificial intelligence Preprint Oct 2026

Scalable AI Uncertainty Quantification via Generalized Laplace Active Subspaces

Reliable uncertainty quantification (UQ) is essential for deploying neural networks in scientific and high-stakes applications, but full Bayesian inference over the network parameters is computationally infeasible. We propose a low-rank generalized Laplace approximation for neural-network UQ based on a small number of...

W. Edeling, P. Coveney · 0 citations
Preprint Oct 2026

Possibility-based Inference: A Deterministic Framework for Robust and Reliable Uncertainty Quantification in Decision-Making

Decision-making under uncertainty remains a key issue in Machine Learning (ML), particularly in the deployment of safety-critical systems, where reliable quantification of system behavior is essential for evaluating behavioral impacts and strengthening confidence in ML models. This contribution, influenced by fuzzy set...

Seyedfakhredin Musavishavazi, M. Alipour · 0 citations

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.