Attention and residual adaptation analysis of node-token GPT2 for traffic flow forecasting

Abstract

Traffic flow forecasting aims to predict future traffic states from historical observations of road networks. Although pretrained language models have demonstrated strong sequence modeling capacity, directly applying GPT-2 to traffic forecasting remains nontrivial because continuous spatiotemporal signals differ substantially from discrete text tokens. To address this issue, this paper proposes a node-tokenized GPT-2 framework and investigates attention adaptation and residual prediction mechanisms for traffic flow forecasting. Specifically, each traffic node is represented as a token by projecting its historical observations into the GPT-2 embedding space. Based on this representation, three attention mechanisms are compared, including causal attention, dense full attention, and dynamic Top-K sparse attention. In addition, a residual prediction strategy is introduced to model future traffic changes relative to the latest observed state. Experiments on the PeMS08 dataset show that GPT2-Causal-Res achieves the best performance, with an MAE of 15.3144, reducing the error by 3.47% compared with the direct prediction baseline. The results indicate that residual prediction improves forecasting accuracy, whereas simply modifying the attention structure does not necessarily lead to reliable modeling of traffic node interactions.