Explainable attention-based intrusion detection for encrypted 5G network traffic

Abstract

The widespread adoption of end-to-end encryption in 5G networks limits the effectiveness of traditional intrusion detection systems that rely on payload inspection. This challenge is particularly critical for detecting Advanced Persistent Threats (APTs), which employ low-rate, long-duration, and stealthy communication strategies to evade conventional defenses. This study presents a privacy-preserving intrusion detection framework that operates exclusively on flow-level traffic metadata without deep packet inspection. Network packets are aggregated into bidirectional flows, from which temporal, statistical, and directional features are extracted to characterize behavioral patterns. A Transformer-based model with self-attention is employed to capture long-range dependencies across encrypted traffic sequences and identify subtle, temporally dispersed attack indicators. The framework is evaluated on a large-scale 5G-relevant dataset containing over one million flow records and compared against classical machine learning, ensemble, CNN, and LSTM models. Results demonstrate high recall and strong F1-score in distinguishing APT from benign traffic. Attention-based and feature-level explanations further reveal that prolonged communication, irregular timing gaps, and directional asymmetry significantly influence detection decisions. The findings support the practicality of explainable Transformer models for secure and scalable APT detection in encrypted 5G environments.