Explainable AI and Machine Learning Framework for Cyber Threat Detection and Adaptive Defense Systems

Abstract

Advanced persistent threats, zero-day exploits, encrypted command-and-control traffic, and botnet campaigns continue to reduce the reliability of conventional intrusion detection systems because static detectors provide limited transparency and weak adaptation under traffic drift. This paper presents an explainable and adaptive machine learning framework that integrates a LightGBM threat detector, SHAP-based decision explanations, density-aware concept drift detection, active incremental updating, and a contextual bandit defense policy. LightGBM is adopted because its leaf-wise gradient boosting structure provides high discrimination for heterogeneous flow features while maintaining low inference latency and native feature-importance support. The framework is evaluated on CIC-IDS2017, UNSW-NB15, and ToN_IoT using stratified train-validation-test splits, leakage prevention, five-run validation, and a 48-hour Kafka-based streaming simulation. The proposed model achieved 99.1% accuracy, 98.7% F1-score, 98.4% recall, and a 0.007 false alarm rate. During streaming evaluation, 14 adaptive model updates reduced mean detection latency from 27.4 s to 11.2 s, while SHAP explanations based on DNS entropy, JA3 rarity, packet interval, and flow-duration evidence reduced analyst triage time by 23%. Comparative results show that the proposed explainable adaptive pipeline improves detection reliability, reduces false alarms, and supports auditable mitigation decisions better than static and black-box IDS baselines.