Invariant Structure Learning with Pre-trained Language Models for Spatio-temporal Graph

Abstract

While Pre-trained Language Models (PLMs) have demonstrated remarkable potential in spatiotemporal forecasting, their deployment in critical real-world systems is hindered by a fundamental vulnerability, brittleness under distribution shifts. Existing PLM-based approaches primarily exploit statistical correlations during training, often by linearizing complex spatiotemporal graph structures into flat token sequences, but fail to distinguish stable mechanisms (e.g., road network topology) from spurious environmental features (e.g., temporary weather). To address this, we propose InvSTG-PLM, an invariance-aware spatiotemporal graph learning framework that grounds PLMs in invariant structures. We formalize an invariance principle via token re-assignment, which simulates environments in the representation space, and design a variance-regularized objective which penalizes predictors relying on variant factors. Guided by this principle, we introduce a practical realization including the InvSTG-Tokenizer and InvSTG-Adapter, which goes beyond standard serialization by acting as an invariant filter: it separates invariant structure from variant factors and aligns invariant geometric structures with the semantic space of PLMs. Theoretical analysis justifies a deterministic robustness bound, showing that mean–variance control upper bounds worst-case risk over simulated environments. Extensive experiments on multiple benchmarks demonstrate that InvSTG-PLM outperforms existing methods and exhibits improved robustness under distribution shifts.