Cloud-Based Security Analytics SIEM Using ELK Stack and Machine Learning

Abstract

The increasing frequency and sophistication of cyberattacks have created significant challenges for organizations in protecting their digital infrastructure. Traditional Security Information and Event Management (SIEM) systems primarily depend on predefined rules and signature-based detection techniques, which are often inadequate for identifying emerging or previously unseen attack patterns. To address these limitations, this paper presents a Cloud-Based Security Analytics SIEM Using ELK Stack and Machine Learning for intelligent cyberattack detection and real-time security monitoring. The proposed framework integrates Elasticsearch, Logstash, and Kibana (ELK Stack) with supervised machine learning algorithms to automate log collection, processing, threat classification, and security visualization. The CICIDS2017 dataset, containing approximately 2.5 million network traffic records, 70 selected features, and 15 attack categories, was used to train and evaluate the intrusion detection models. Random Forest, LightGBM, and XGBoost classifiers were developed and compared using multiple performance metrics. Experimental results indicate that the XGBoost model achieved the best overall performance with an accuracy of 99.86% and a Macro F1-score of 89.57%, demonstrating its effectiveness in multiclass cyberattack classification. The trained model was integrated with the ELK-based analytics platform to support automated attack prediction and centralized visualization through Kibana dashboards. The proposed system provides an efficient, scalable, and intelligent approach for enhancing cybersecurity monitoring by combining cloud-based log analytics with machine learning-driven threat detection. The framework can assist Security Operations Centers (SOCs) in improving incident detection, accelerating security analysis, and supporting proactive cyber defense.