Privacy-Preserving Zero-Day Malware Detection in Encrypted Traffic Using Temporal-Structural Transformers

Abstract

End-to-end encryption protocols such as TLS 1.3 and QUIC strengthen user privacy but reduce the visibility available to conventional payload-based network-security inspection. This study presents a privacy-preserving Temporal-Structural Transformer (TST) framework for zero-day malware detection in encrypted network traffic without decrypting packet payloads. The framework combines transformer-based temporal modelling of packet sequences with graph attention mechanisms for capturing structural relationships among flows. Observable metadata, including packet size, direction, inter-arrival time, flow statistics, and available handshake attributes, are used to represent encrypted communication behavior. A contrastive-learning objective is incorporated with classification loss to improve generalization to previously unseen malware families. The framework is evaluated using multiple benchmark datasets containing TLS 1.2, TLS 1.3, and QUIC traffic, with zero-day testing, ablation analysis, robustness testing, cross-validation, and real-time performance assessment. The proposed model achieves an F1-score of 0.93 in the overall evaluation and 0.934 in the zero-day setting, with a zero-day ROC-AUC of 0.95. Ablation results indicate contributions from the temporal, graph, and contrastive-learning components, while robustness tests show limited degradation under the evaluated perturbations. Reported inference latency is below 10 ms per flow, with throughput above 10,000 flows per second. These findings support metadata-based temporal and structural modelling as a privacy-preserving approach to zero-day malware detection in encrypted traffic.