A Representation-and-Scoring Framework for Zero-Day Malicious Traffic Detection Using Self-Supervised Graph Learning

Abstract

Recent network traffic detection systems tend to perform well when identifying known attacks, yet their effectiveness drops significantly when faced with previously unseen malicious behavior. This issue becomes particularly important in zero-day scenarios, where predefined signatures and attack- specific patterns are either unavailable or unreliable. In this study, we explore a graph-based approach for zero-day malicious traffic detection. Network flows are first converted into traffic graphs, then encoded using self-supervised representation learning, and finally evaluated through post-hoc distance-based scoring methods. Instead of limiting the analysis to a single benchmark, we examine the proposed framework across three widely used datasets—USTC-TFC2016, CTU-13, and CIC-IDS2017—while focusing on key factors such as scorer design, graph construction, sampling strategy, and robustness under different evaluation settings. The proposed framework integrates self-supervised graph representation learning with lightweight zero-day scoring techniques, including centroid distance, Mahalanobis distance, and k-prototype minimum distance. The results indicate that the choice of scoring method has a significant impact on generalization performance. On USTC-TFC2016, the framework performs particularly well, with k-prototype scoring achieving the best overall results, while Mahalanobis distance also remains highly competitive. For CTU- 13, the findings indicate that mixed-traffic graph construction poses notable challenges; however, applying chunk purity filtering yields a clear performance improvement. In the case of CIC-IDS2017, adopting train- aware balanced chunk sampling enhances both overall accuracy and consistency across folds, suggesting that graph construction and sampling strategies play a role as important as the encoder itself in achieving reliable zero-day detection. Across all three datasets, a consistent pattern emerges: graph-based zero-day detection is both feasible and effective, but its success strongly depends on how traffic is segmented, represented, and evaluated. Additional ablation studies—covering training epochs, chunk size, number of sampled graphs, and random seed variations—further confirm the robustness of the proposed approach. Notably, increasing training depth proves beneficial, whereas simply increasing the number of sampled graphs per capture does not necessarily yield better results. Overall, this work presents a practical graph- based pipeline for zero-day traffic detection and provides a detailed empirical analysis of the conditions under which such methods can generalize effectively across diverse and heterogeneous malicious traffic datasets.