Encoder-Level Temporal Fusion Transformers for Robust Multi-Object Tracking

Abstract

Effective temporal reasoning is essential for robust multi-object tracking (MOT), especially in dynamic and crowded scenes where frequent occlusions, appearance ambiguity, complex object interactions, and abrupt motion occur. Although recent Transformer-based trackers have achieved impressive results, most existing approaches introduce temporal information mainly at the decoder stage, after spatial features have already been encoded. This late temporal reasoning design limits the influence of historical context on feature representation, often leading to unstable trajectories and identity switches under challenging motion conditions. In this paper, we propose Gated Temporal Fusion (GTF), an encoder-level temporal reasoning framework that integrates historical tracklet information directly into the Transformer encoder. GTF incorporates a reliability-aware temporal memory and a dual cross-attention mechanism to align historical and current features, followed by an adaptive gating module that selectively fuses temporal cues at each spatial location. To prevent degenerate temporal fusion, we introduce an entropy-based regularization that alleviates low-diversity gating behavior without requiring explicit gate-value supervision. Experiments on three challenging benchmarks—MOT17, DanceTrack, and SportsMOT—show that GTF improves identity association and overall tracking robustness, particularly under fast motion, appearance ambiguity, and frequent occlusions. Quantitatively, under the reported benchmark settings, GTF improves association-related metrics over the corresponding MOTIP baselines, including + 1.7 AssA and + 0.9 IDF1 on MOT17, + 2.0 HOTA and + 2.5 AssA on DanceTrack compared with the reproduced MOTIP baseline, and + 1.5 HOTA and + 1.6 AssA on SportsMOT compared with the MOTIP reference using extra data. Ablation studies further support the roles of encoder-level temporal fusion, adaptive gating, entropy regularization, and memory update strategies, highlighting the advantages and practical trade-offs of early reliability-aware temporal integration for robust multi-object tracking.