Skip to content

Marginal Log-Likelihood Increments under Dirichlet-Smoothed Markov Estimation

Sep 2026 · arXiv.org · Vol abs/2609.25675 · 0 citations · 11 references
Computer Science

TL;DR

An upper bound on the gain available to any acquisition is obtained, which expresses a millinat difference as a share of what is attainable as a share of what is attainable in a Dirichlet-smoothed transition model.

Abstract

For a Dirichlet-smoothed transition model, the effect of adding one workflow trace to the training archive is an exact change in reference-weighted log likelihood. We derive that change and show that it is a weighted reduction of Kullback--Leibler divergence between the reference conditionals and the model. From this form we obtain an upper bound on the gain available to any acquisition, which expresses a millinat difference as a share of what is attainable, an exact covariance identity for the effect of the reference weighting, and a sign criterion for the interaction between two candidates, from which the batch objective is neither submodular nor supermodular. A case study on the BPI Challenge 2012 loan-application log measures all three and finds a positive selection result in one of the four combinations of reference weighting and budget unit. There, of two regressors fitted to identical descriptors and identical labels, the one that predicts individual increments far more accurately, median $R^2$ 0.87 against 0.62, realizes the smaller share of the attainable gain, 61 against 69 per cent, so ranking accuracy for individual traces is neither necessary nor sufficient for batch quality.

View source

Similar papers

Preprint Oct 2026

Beyond the Kullback--Leibler divergence: the likelihood-ratio distribution as a detectability measure

The Kullback--Leibler (KL) divergence is the expectation value of the log-likelihood ratio between two specified models, evaluated with respect to one of the two models. It provides an average measure of the distinguishability of the models. We examine its connection to discovery potential and the accuracy of parameter...

Amirhossein Samandar, M. Barandiaran, Lin-Nan Lu et al. · 1 citation

Generalized Score Matching for Parameter Estimation on Convex Domains

This work derives the generalized score matching objective on a convex subset of $\mathbb{R}^{d}$ constructively starting from Minimum Probability Flow (MPF) learning, and shows how classical score matching as well as domain-adapted variants for non-negative data arise naturally within the proposed framework.

Nishanth Shetty, Saisuchith Mahajan, C. Seelamantula · 0 citations
#machine learning Preprint Sep 2026

Exact information accounting for SGD methods

An exact, information-theoretic analysis of stochastic gradient descent and its variants is given, showing that a preconditioned SGD step is the posterior-mean update of a Gaussian Bayes model, and that its one-step regret splits into an intrinsic-time cost and a change in comparator information.

Akshay Balsubramani · 0 citations
#machine learning Preprint Oct 2026

An Accuracy-Information Tradeoff for Loss-Difference Conditional Mutual Information

Loss-difference conditional mutual information (ld-CMI) uses the smallest of the standard observations in the supersample hierarchy of generalization bounds: it measures what a learner's loss differences reveal about which candidate of each pair it was trained on. Accuracy is known to force information into the model;...

Hazar Yueksel · 0 citations

The cost of useful natural gradient updates

What information is needed to turn a natural-gradient direction into a useful finite update? Under a population Kullback-Leibler (KL) budget, we call a step useful if it is feasible and loses at most a fraction $\varepsilon$ of the best feasible gain along the direction. We construct a four-state exponential family who...

Subhransu S. Bhattacharjee, Dylan Campbell, Rahul Shome · 0 citations
#machine learning Preprint Oct 2026

The Impact of Likelihood Tempering on the Limiting Predictive Moments of Variational Bayesian Linear Neural Networks

In wide Bayesian neural networks, Gaussian mean-field variational inference is prone to"prior dominance": the Kullback-Leibler (KL) regularization term of the ELBO outweighs the expected log-likelihood, and the variational predictive distribution collapses to the prior predictive as the width $M$ grows. Tempering the l...

Ian Zhang, Thibault Randrianarisoa · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.