SHAP driven explainable machine learning for malware classification: From detection to explanation

Abstract

Detecting malware is a critical aspect of cybersecurity, focused on identifying and preventing malicious software from compromising computer systems. The increasing complexity and sophistication of modern malware often render conventional detection techniques inadequate, necessitating more advanced approaches. In this study, machine learning (ML) algorithms, known for analysing patterns and behaviours within large datasets, were systematically applied for malware detection using the publicly available EMBER dataset. This work provides a comprehensive benchmark of classical ML models on the robust, large-scale dataset, offering crucial insights into their performance ceilings and practical trade-offs in real-world applications. The primary objective was to evaluate the effectiveness of ML algorithms in distinguishing between benign and malicious files through binary classification. The results highlighted that the LightGBM (LGBM) classifier achieved the highest detection accuracy of 0.9858% ± 0.0003% and an Area Under the Curve (AUC) value of 0.99. The robustness and reliability of the models were further validated using evaluation metrics such as precision, recall, and F1-score. The findings highlight the significant potential of classical ML and cross validation in malware detection to address the evolving threat landscape. An Explainable AI (XAI) approach based on SHAP (SHapley Additive explanations) has been employed to enhance the interpretability and transparency of the machine learning models. This technique analysed feature importance and provided global explanations for individual malware classification predictions. Model decisions were interpreted, improving trust and insight into how different features influence classification outcomes