From Machine Learning to LogBERT: A Survey of Web Log Anomaly Detection for APT and Botnet Threats

Abstract

  Automated log analysis has become a cornerstone of cybersecurity operations, yet research on log-based anomaly detection remains scattered across different modalities, learning paradigms, and threat models. This survey brings together work on log anomaly detection-spanning signature-based and classical unsupervised methods, deep sequential models, and BERT/LogBERT self-supervision-with a particular focus on botnet and advanced persistent threat (APT) identification in web server environments, using Black Energy as a representative APT/botnet case study. Prior work is organised into four categories: supervised, unsupervised, deep learning, and hybrid/BERT-based approaches. We compare the datasets, underlying assumptions, and limitations across these categories, and identify a specific research gap at the intersection of LogBERT-style self-supervision, session-aggregated web access logs, and Black Energy-class behavioral indicators. Three problems in particular stand out: a mismatch between methodology and data modality, a lack of threat-specific evaluation, and the absence of public benchmarks-each of which is substantiated in detail in Section 4.7. This paper's contribution is threefold: a comprehensive literature synthesis covering more than 75 peer-reviewed studies published between 2016 and 2025, organised into the four methodological categories described above; a master synthesis matrix (Table 7, split as Tables 7a–7e) summarizing the qualitative advantages and limitations of each approach; and a formal gap analysis that confines its claims strictly to the reviewed corpus. No empirical results, new detection methods, or experimental validation are presented here. Instead, this work aims to situate future empirical research within the gaps it identifies.