Malware Analysis Using Data Mining Techniques: A Review of Static, Dynamic and Hybrid Analysis

Abstract

Malware continues to evolve through obfuscation, packing, polymorphism, and other evasion techniques, challenging traditional detection approaches. Data mining (DM) and machine learning (ML) is become important by the day for identifying malicious software through static characteristics, runtime behaviour, and combinations of both. This review examines 46 selected studies on malware detection using DM and ML, with emphasis on static, dynamic, and hybrid analysis. The selected literature is synthesized according to malware platforms, datasets, feature representations, learning algorithms, reported performance, and methodological limitations. The review indicates that static analysis provides computationally efficient detection based on features such as PE headers, strings, byte sequences, and opcode representations, whereas dynamic analysis provides behavioural information through API calls, system calls, memory traces, and execution activities. Hybrid approaches combine complementary feature sources and have demonstrated promising detection performance across several studies. Random Forest, Support Vector Machine, Decision Tree, and ensemble-based approaches are frequently reported, while deep learning approaches are increasingly explored for complex feature representations. However, comparison across studies is constrained by differences in datasets, class distributions, feature engineering procedures, and evaluation protocols. Recurring limitations include small or imbalanced datasets, limited real-world validation, insufficient evaluation against unseen malware, computational overhead, and limited model interpretability. Based on these observations, the review identifies explainable artificial intelligence, transfer learning, lightweight real-time detection, and temporal evaluation as important directions for future research.