HyTMTC: A Pre-Training Method for Multi-Scenario Network Traffic Classification With Hybrid Transformer-Mamba

Abstract

Network traffic classification is central to security monitoring and network management in heterogeneous environments. Existing deep learning approaches are often trained for a single scenario and require large amounts of labeled data, making them difficult to reuse when applications, traffic types, or encryption settings change. We present HyTMTC, a pre-training and fine-tuning framework for multi-scenario network traffic classification. HyTMTC encodes each flow as a unified multimodal token sequence that combines raw bytes, packet length, and direction, allowing protocol traces and communication behavior to be modeled together. To match the one-dimensional nature of traffic data, HyTMTC adopts a hybrid Transformer-Mamba backbone. Mamba captures contiguous byte- and packet-level patterns, while Transformer attention strengthens interactions across non-adjacent fields and packets. A self-attention fusion module further integrates the multimodal representations during fine-tuning. This design improves traffic representation without relying on scenario-specific feature engineering. Experiments on seven public datasets show that HyTMTC achieves an average F1-score of 95.03% and outperforms nine representative baselines. It also remains effective in encrypted, VPN, and few-shot settings, and maintains the ability to detect unknown attacks. These results demonstrate its effectiveness and stability for multi-scenario network traffic classification.