Skip to content

Temporal Heterogeneous Graph Pretraining for Relational Deep Learning

Sep 2026 · arXiv.org · Vol abs/2609.35219 · 0 citations · 31 references
Computer Science

TL;DR

This framework combines Multi-scale Time Encoding, which captures record age using learnable time scales and type-specific projections, with Rotary Time Encoding, which represents signed inter-record intervals through rotary transformations during graph propagation.

Abstract

Relational deep learning models database rows and foreign-key links as a heterogeneous graph for prediction from record attributes and relational context. These graphs contain two distinct temporal signals: record age changes with the prediction cutoff, while intervals between observed records remain fixed. Prior work often treats time as a single signal or studies temporal representation and pretraining separately. We investigate how explicitly encoding both signals affects temporal pretraining for downstream tasks. Our framework combines Multi-scale Time Encoding, which captures record age using learnable time scales and type-specific projections, with Rotary Time Encoding, which represents signed inter-record intervals through rotary transformations during graph propagation. We pair these encodings with three self-supervised objectives: historical relation recovery, horizon-aware future relation activity prediction, and temporal subgraph contrast. All inputs respect their observation cutoffs. Pretraining proceeds in two stages: subgraph contrast first learns neighborhood representations, followed by refinement through either relation recovery or future activity prediction. We evaluate on five RelBench datasets across 11 classification and regression tasks using heterogeneous GNN and graph Transformer backbones. With both encodings, the best evaluated staged schedules improve over supervised training with the same encodings by 3.02% and 1.06% on the two backbones, respectively, and over controls without pretraining or either encoding by 3.24% and 2.37%.

View source

Similar papers

#machine learning Preprint Oct 2026

Revisiting Simple Baselines for Relational Deep Learning

Relational deep learning has emerged as a powerful paradigm for learning end-to-end predictive models over relational databases by representing them as temporal heterogeneous graphs. Recent advances based on graph neural networks and Transformers have achieved strong performance across diverse domains. However, the eff...

Viet T. Nguyen, Vijay Prakash Dwivedi, Gunther Gust · 0 citations
#artificial intelligence Preprint Oct 2026

Learning Across Temporal Graph Representation Silos

Temporal graph data are recorded in different representations, but whether these differences also determine the scope of learned knowledge reuse remains unclear. This study investigates whether predictive structure learned from relational histories can be reused between continuous-time dynamic graphs (CTDGs) and tempor...

Min-Woo Yu, Young-Guk Ha · 0 citations
Conference Open access Sep 2026

History Doesn’t Repeat, but Its Patterns Echo: A Parallel Pairwise Negative-Sampling Framework for Temporal Link Prediction

This paper proposes ATNSF, a temporal graph learning framework with a hybrid negative sampling strategy that uses a portion of historical edges as hard negatives and introduces a pairwise sampled softmax loss that contrasts each positive instance with a batch of negatives to learn more discriminative representations.

Yong-Chun Jiang, Heng Zhang, Jian Gao et al. · 0 citations
Sep 2026

Spectral-temporal dual view graph convolutional networks: A deep learning framework for dynamic bipartite graphs.

Dynamic bipartite graphs (DBGs) are widely used in real-world scenarios, where representation learning is particularly challenging due to the heterogeneity of node types and the temporal evolution of interactions. A key difficulty lies in jointly capturing non-stationary micro-level preference dynamics and macro-level...

Zhe-Zhe Xing, Yu-Xin Ye, Zi-Heng Li et al. · 0 citations
#machine learning Preprint Oct 2026

To Learn is to Wander: Learning Across Graphs and Tasks with Random Walks

Wander is proposed, a graph foundation model designed to operate across graphs, feature spaces, relational schemas, and prediction tasks within a single pretrained checkpoint that preserves performance in specialized settings while enabling positive transfer and the composition of separately learned capabilities at inf...

Louis Tichelman, Xing-Yue Huang, Jinwoo Kim et al. · 0 citations
#artificial intelligence Preprint Oct 2026

TAM: Task-Aware Memory Distillation for Efficient Spatiotemporal Prediction

Knowledge distillation enables efficient spatiotemporal prediction by transferring knowledge from an accurate teacher to a compact student. However, matching outputs or features independently for each sample leaves cross-sample predictive structure underused. Exploiting this structure requires representations and histo...

Yu-Qi Li, Xiaoqin Feng, Fan Xu et al. · 0 citations

Related blog posts

MIT News · Artificial Intelligence Nov 17, 2026

Learning the language of machine learning

By studying the computational foundations of language learning, Associate Professor Jacob Andreas builds intelligent systems that can communicate effectively with humans.

Microsoft Research Blog Nov 16, 2026

TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient

AI coding agents are increasingly help developers fix bugs and build features largely on their own, but every interaction with their models consumes tokens. In each session, an agent sends instructions, tool descriptions, and session history to its model. Most efforts to reduce this cost focus on individual tasks. Yet developers return to the same […] The post TokenEconomizer: Learning from Experience to Make AI Coding Agents More Efficient appeared first on Microsoft Research.

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.