Skip to content

Edge Accuracy Is Not Enough: Why Dynamics-Learned Structure Fails to Transfer to Inverse Problems

Oct 2026 · 0 citations · 14 references
Computer Science

TL;DR

It is proved that approximate structure provides estimation-error benefits whenever the edge error satisfies $\Delta<n^2 - kn$, reducing sample complexity from $O(n^2)$ to $O(kn+\Delta)$.

Abstract

A natural strategy for inverse problems with scarce labelled data is to transfer relational structure learned from abundant forward-simulation data. We show this strategy fails systematically, even when it satisfies the standard theoretical justification for why structure should help. We prove that approximate structure provides estimation-error benefits whenever the edge error satisfies $\Delta<n^2 - kn$, reducing sample complexity from $O(n^2)$ to $O(kn+\Delta)$. Structure learned via Neural Relational Inference (NRI) from dynamics prediction satisfies this condition, yet on a source-localisation task across 180 CFD-simulated hydrogen-leak scenarios and 180 acoustic scenarios, it degrades performance by 116% and 201% relative to a flexible, task-optimised attention baseline, while a physics-based prior (Green's function) degrades by only 69-72%. Four independent lines of evidence show this is not a tuning failure: NRI improves only 0.5% when given 18x more training data (versus 16.6% for the task-optimised baseline, $p<0.001$); performance is insensitive to the NRI edge threshold across a wide range; the dynamics-learned graph overlaps the task-optimal graph on only 6% of edges; and two further dynamics-derived structure estimators (correlation- and mutual-information-based) show no measurable benefit over a structure-free baseline, with the correlation-based estimator performing markedly worse. We formalise this gap as a statement about approximation error that the edge-accuracy condition cannot control, and we provide a lightweight transferability test (Jaccard similarity against a partially-observed target-task graph) that separates successful from failed transfer in all four domain/structure pairs we evaluate, using under an hour of computation and 15-20% of target-domain data; we present this as a heuristic calibrated on few cases, not a validated general threshold.

View source

Similar papers

#machine learning Preprint Oct 2026

Pretraining Shapes Spectral Structure: Architecture- and Strategy-Conditional Prediction of OOD Robustness in Foundation Models

Can we determine whether a foundation model will generalize out-of-distribution (OOD) before any target data is available? Existing diagnostics require source or target data, which rules them out before a target domain exists. Those that use the weights alone apply one statistic to every architecture, and do not separa...

Sang-Peel Bae, Sk. Miraj Ahmed, Shinjae Yoo et al. · 0 citations
#machine learning Preprint Oct 2026

Learn Only Where You Can't Compute: Structured Corrections to Gaussian Process Posteriors

Physics-informed Gaussian processes (GPs) provide a principled framework for scientific inference by expressing uncertainty over empirical measurements, boundary conditions, and governing equations. However, they struggle to scale to larger datasets and nonlinear information. Conversely, diffusion models excel at learn...

Tobias Weber, Carla Sagebiel, Philipp Hennig · 0 citations
#machine learning Preprint Oct 2026

Do ResNets Route? Sparse Interaction Experts in Residual Networks

Residual networks execute every block for every input, yet their functional contributions need not be input independent. We formulate a trained ResNet as a set function over binary residual-branch masks and apply M\"obius inversion to decompose its output exactly into individual residual corrections and higher-order in...

Liang Yan, Si-Ying Chen, Kai-Jie Chen et al. · 0 citations
#machine learning Preprint Oct 2026

Adaptive Mean Estimation by In-Context Learning: A Gradient-Flow Analysis

This work studies how statistical adaptivity is learned in a controlled location-estimation problem, and combines attention experts through either a softmax mixture of experts or a gated linear unit (GLU), and analyze stagewise gradient flow.

Martin Eppert, K. Balasubramanian, Subhro Ghosh et al. · 0 citations
#artificial intelligence Preprint Oct 2026

TICDA: Tabular In-Context Data Attribution

Tabular foundation models (TFMs) achieve strong predictive performance by conditioning on labeled demonstrations provided in context, without any parameter update. Yet how individual demonstrations shape a given prediction remains poorly understood. This gap matters in practice: the context is often assembled from what...

Yacine Benihaddadene, Milan Bhan, Eliot Dugelay et al. · 0 citations

Let It Go or Learn to Self-Correct: Continuous Diffusion for Constrained Discrete Tasks

The results suggest that continuous diffusion models can learn nontrivial global constraints, but discrete reasoning tasks require better alignment between training and inference: either through samplers that reduce commitment to early decisions, or through training that teaches the model to correct its own inference-t...

M. Drozdova, S. Nguyen, François Fleuret · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.