Skip to content

Context Window Failures in Relational Foundation Models

Aug 2026 · 0 citations · 5 references
Computer Science

Abstract

Recent Relational Deep Learning architectures have been proposed as foundation models for multi-table relational data, yet they impose constrained neighborhood budgets that force row truncation when an entity has many related records. We introduce Animus, a synthetic financial dataset in which predicting customer income requires aggregating up to tens of thousands of transactions. On the raw representation, three recently proposed models (RT, Griffin, RelGT) achieve $R^2 \le 0.18$; a single, routine, temporal pre-aggregation step recovers $R^2$ up to $0.65$. This questions whether current relational foundation models are ready for high-cardinality real-world data.

View source

Similar papers

Preprint Aug 2026

MINT: A Universal Zero-Shot Predictor for Transaction Data

The Multimodal Instruction Network for Transactions (MINT), a framework that connects a pretrained transaction sequence encoder to a decoder-only LLM through lightweight embedding injection, transaction-language alignment, and instruction tuning, achieves state-of-the-art predictive question-answering performance in bo...

Parameswaran Kamalaruban, Viktor Drobnyi, Maeve Madigan et al. · 0 citations
#machine learning Preprint Aug 2026

Can LLMs Use Relational Transformer Embeddings?

It is argued that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

Francisco Galuppo Azevedo, Clarissa Lima Loures · 0 citations
Preprint Aug 2026

Incremental Evaluation and Training in Relational Deep Learning

Relational Deep Learning (RDL) models multi-tabular databases as temporal heterogeneous graphs to enable end-to-end representation learning. However, prevailing RDL evaluation practices rely on static, single-episode dataset snapshots, overlooking the continuous, time-evolving nature of real-world databases. Consequent...

Jakub Peleška, Gustav Šír · 0 citations

ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations

ReLaG is introduced, a modality-agnostic framework that models sample relatedness through a hierarchical latent-variable process and infers groups of related samples using proximity graphs and community detection to produce independent train--test subsets.

Anthony Lavertu, Jacob A. Cote, S. Gobeil et al. · 0 citations

Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

This framework combines Multi-scale Time Encoding, which captures record age using learnable time scales and type-specific projections, with Rotary Time Encoding, which represents signed inter-record intervals through rotary transformations during graph propagation.

Yi-Xin Peng, Er Jin, Diego Collarana et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.