Skip to content

ALICE: In-context, Zero-shot, Mutual Information Estimation

Sep 2026 · arXiv.org · Vol abs/2609.34962 · 0 citations · 85 references
Computer Science

TL;DR

For the first time, it is shown that a single model closes the gap with neural estimators trained separately for each distribution, while natively supporting different data dimensionality and sample cardinality, enabling zero-shot MI analysis across scientific domains.

Abstract

Estimating mutual information (MI) from samples is a central objective in a variety of scientific fields. Modern neural estimators are accurate in the large-data regime, but they fall short when data is scarce, and each must be fit anew for every distribution under study. Current estimators are moreover tied to specific data types. These constraints limit their adoption in many applications where per-distribution training is impractical and sample sizes are small. We present ALICE, a foundation model that removes per-distribution training, while achieving competitive estimation accuracy. Trained exclusively on a broad family of synthetic distributions, ALICE acts as an in-context estimator of rectified-flow velocity fields: conditioned on samples of an unseen distribution, it estimates that distribution's velocity field without any explicit training. MI is then obtained through a fixed identity that integrates the squared difference between the joint and conditional fields. We validate ALICE on a standard, challenging benchmark and apply it in three domains, biology, genetics, and neuroscience, whose data the model has never seen. For the first time, we show that a single model closes the gap with neural estimators trained separately for each distribution, while natively supporting different data dimensionality and sample cardinality, enabling zero-shot MI analysis across scientific domains.

View source

Similar papers

Open access Sep 2026

Sample-Conditional Mixtures of Entropy Estimators for Short Sequences Under a Uniform Marginal Null

The adaptive sample-conditional entropy diagnostic (ASED), in which a compact network trained offline maps a frequency-of-frequencies descriptor of the sample to convex mixture weights over six classical estimators, is introduced, and an oracle inequality bounding the excess risk of such a mixture by the L1 error of it...

Guillermo Sosa-Gómez · 0 citations
#machine learning Preprint Oct 2026

Two-Sample Testing via Generative Processes

Deciding whether two samples come from the same distribution is a classical problem in statistics, and generative transport offers a new way to approach it. We build a stochastic interpolant directly between the two samples and observe that, for a symmetric schedule, its law is invariant under the time reflection $t \m...

Eshant English, Kenji Fukumizu, Taiji Suzuki · 0 citations
Preprint Sep 2026

Universal Estimation of the Fisher Information for Processes with Memory

The Fisher information matrix (FIM) determines the accuracy attainable in an estimation problem. For a parametric process with memory, it has a closed form only for narrow classes of models, and it cannot be computed when the model is available only as a simulator. This paper estimates the Fisher information rate of su...

Yi-Bo Shi, Cristian R. Rojas · 1 citation
#machine learning Preprint Oct 2026

An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information

Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on. Accuracy is known to force information into the model;...

Hazar Yueksel · 0 citations
#machine learning Preprint Oct 2026

Adaptive Mean Estimation by In-Context Learning: A Gradient-Flow Analysis

This work studies how statistical adaptivity is learned in a controlled location-estimation problem, and combines attention experts through either a softmax mixture of experts or a gated linear unit (GLU), and analyze stagewise gradient flow.

Martin Eppert, K. Balasubramanian, Subhro Ghosh et al. · 0 citations
Preprint Oct 2026

Poisson empirical Bayes estimation of sums of random variables via minimum-distance methods

The estimation of sums of functions of observable and unobservable variables is a long-standing problem in statistics, with applications in many domains. We consider this problem in Poisson mixture models, where empirical Bayes provides a natural framework but nonparametric theory remains limited. We develop a nonparam...

S. Favaro, S. Fortini, Soham Jana · 1 citation

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.