Skip to content

Context Window Failures in Relational Foundation Models

Aug 2026 · 0 citations · 5 references
Computer Science

Abstract

Recent Relational Deep Learning architectures have been proposed as foundation models for multi-table relational data, yet they impose constrained neighborhood budgets that force row truncation when an entity has many related records. We introduce Animus, a synthetic financial dataset in which predicting customer income requires aggregating up to tens of thousands of transactions. On the raw representation, three recently proposed models (RT, Griffin, RelGT) achieve $R^2 \le 0.18$; a single, routine, temporal pre-aggregation step recovers $R^2$ up to $0.65$. This questions whether current relational foundation models are ready for high-cardinality real-world data.

View source

Similar papers

#artificial intelligence Preprint Sep 2026

MetaRTL: Meta-path Attention Enhanced Relational Table Learning

Relational table learning has gained increasing attention with the widespread use of relational databases. Existing methods typically rely on deep GNN or HGNN stacks, leading to high computational costs and limited performance on large real-world databases. We propose MetaRTL, a two-stage framework for scalable and exp...

Ken Zhong, Wei-Chen Li, Zheng Wang · 0 citations
Preprint Aug 2026

MINT: A Universal Zero-Shot Predictor for Transaction Data

The Multimodal Instruction Network for Transactions (MINT), a framework that connects a pretrained transaction sequence encoder to a decoder-only LLM through lightweight embedding injection, transaction-language alignment, and instruction tuning, achieves state-of-the-art predictive question-answering performance in bo...

Parameswaran Kamalaruban, Viktor Drobnyi, Maeve Madigan et al. · 0 citations
#machine learning Preprint Aug 2026

Can LLMs Use Relational Transformer Embeddings?

It is argued that soft-token fusion requires stronger alignment objectives and schema-aware design before it can serve as a reliable route to relational prediction.

Francisco Galuppo Azevedo, Clarissa Lima Loures · 0 citations
Preprint Aug 2026

Incremental Evaluation and Training in Relational Deep Learning

Relational Deep Learning (RDL) models multi-tabular databases as temporal heterogeneous graphs to enable end-to-end representation learning. However, prevailing RDL evaluation practices rely on static, single-episode dataset snapshots, overlooking the continuous, time-evolving nature of real-world databases. Consequent...

Jakub Peleška, Gustav Šír · 0 citations
#artificial intelligence Preprint Sep 2026

ReLaG: A Scalable Framework Generalizing Random Splits to Data with Latent Relations

Random splitting can yield non-independent train--test subsets when a dataset contains related samples, as is common in certain applications such as biochemical studies. This leads to overly optimistic generalization estimates. Here, we introduce ReLaG, a modality-agnostic framework that models sample relatedness throu...

Anthony Lavertu, Jacob A. Cote, S. Gobeil et al. · 0 citations
#machine learning Preprint Sep 2026

Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

Relational deep learning models database rows and foreign-key links as a heterogeneous graph for prediction from record attributes and relational context. These graphs contain two distinct temporal signals: record age changes with the prediction cutoff, while intervals between observed records remain fixed. Prior work...

Yi-Xin Peng, Er Jin, Diego Collarana et al. · 0 citations

Related blog posts

Microsoft Research Blog Sep 30, 2026

Forecasting space weather risks on power grids

Extreme space-weather events can damage power systems on Earth and degrade GPS accuracy and satellite operations. A new machine learning system can predict where damage is likely to occur 30-60 minutes before a storm arrives. The post Forecasting space weather risks on power grids appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.