Machine Learning in Cyber Security: A Systematic Literature Review of Intrusion Detection, Malware Analysis, and Adversarial Robustness

Abstract

Abstract The escalating scale and sophistication of cyber threats - including advanced persistent threats, ransomware, and zero-day exploits - has driven a decade-long shift in cybersecurity research away from static, signature-based defences and toward machine learning (ML) systems capable of learning attack patterns from data and generalising to previously unseen threats. This paper presents a systematic literature review of machine learning in cyber security, organising the field into classical supervised intrusion detection, deep learning and hybrid detection architectures, graph-based and provenance-aware detection, machine-learning-based malware and phishing detection, and the adversarial machine learning literature that studies how the very ML systems built to defend networks can themselves be attacked and manipulated. The review examines the benchmark datasets - including NSL-KDD, UNSW-NB15, and CICIDS2017 - and evaluation practices that recur across this literature, situates the technical literature against industry data on the real-world cost and frequency of breaches, and discusses cross-cutting challenges including class imbalance, concept drift, adversarial vulnerability, explainability, and the operational gap between benchmark accuracy and production deployment. The paper concludes by proposing a conceptual layered framework for machine-learning-based security operations that integrates detection, adversarial-robustness testing, and human-analyst oversight, and by outlining directions for future research. Keywords: machine learning, cyber security, intrusion detection, malware detection, adversarial machine learning, phishing detection, network security