An Adaptive Explainable Hybrid Data Mining Framework for Zero-Day Cyber Threat Detection Using Network Traffic Analysis

Abstract

Fast development of network infrastructures brought about increasing amount of network traffic and its diversification resulting in diversified possibilities while disclosing inability of traditional Intrusion Detection Systems (IDS) to cope with modern types of threats like Zero-Day, hi-tech attacks. In this paper there is presented a solution to this problem through the means of dossier mining and Adaptive Explainable Hybrid Data Mining Framework. Instead of traditional static, domain-dependent security models there is proposed to implement a data stream mining, active feature extraction and machine intelligence in explainable way. An unsupervised dossier mining stage is implemented using Isolation Forest algorithm as the first layer of anomaly detection and outliers’ investigation. It is complemented with a second stage of composite directed ensemble consisting of assembling three algorithms (XGBoost, LightGBM, and Random Forest) which constitute strong models for danger classification. In order to mine knowledge about time-dependent behavior patterns an adaptive feature selection approach is implemented by investigating extreme-spatial and clearly evolving network dossiers. Additionally, in order to solve the problem of black-box approach of complex dossier mining models an Explainable AI (XAI) layer was developed with usage of SHAP (Shapley Additive Explanations). Such an approach allows mining feature attributions and restrict them in order to make model interpretable from the viewpoint of humans explaining the rationales for each classification of traffic pattern and thus providing an early warning system for unknown attacks. Proposed solution was tested and compared to the traditional approaches on two benchmark datasets CSE-CIC-IDS2018 and UNSW-NB15. Results of experiments showed the efficiency of the framework achieving very high detection accuracy of 98.7% and F1 score of 98.2% for unknown attacks with significant drop in false positive rate and inference latency suitable for real-time streaming processing. Thus, the research demonstrates that using adaptive model with dossier mining approach, along with effective network signatures and model interpretability, a resistant and flexible protection measure may be implemented.