Network log anomaly detection and evaluation based on unsupervised deep learning

Abstract

With the rapid development of Internet technology and the deepening of digital transformation, network log data has experienced explosive growth, and network attacks are becoming increasingly complex and covert. Traditional manual analysis and detection methods, which rely on labeled data and ignore temporal and semantic features, are no longer sufficient to meet the needs of modern network security. This study aims to construct an efficient unsupervised deep learning framework for network log anomaly detection, addressing the problems of scarce anomaly samples, difficulty in labeling, and insufficient detection performance of traditional methods. First, network log data undergoes preprocessing such as cleaning, integration, and feature extraction, employing hybrid sampling to balance data distribution. Then, a model combining an autoencoder and generative adversarial network (GAN) is constructed, and hyperparameters are adjusted through intelligent optimization algorithms. Anomalies are identified using a dual mechanism of reconstruction error and discriminant loss. Finally, the model performance is validated on three public datasets—HDFS, BGL, and Thunderbird— based on multiple metrics including accuracy, F1 score, and AUC. Experimental results show that the proposed framework's autoencoder model achieves an F1 score of 0.92 and an AUC of 0.95, significantly outperforming the GAN model and traditional methods. It effectively balances detection accuracy and recall while reducing false positives. This research does not rely on a large amount of labeled data and can automatically identify potential cybersecurity threats, providing intelligent and automated monitoring methods for cybersecurity operations and promoting the practical development of unsupervised deep learning in the field of anomaly detection.