Explainable Malware Detection from Noisy API Sequences with RAG-Based MITRE ATT&CK Mapping

Abstract

As sophisticated evasion techniques like polymorphism and staged execution increasingly neutralize conventional signature-based defenses, dynamic API sequence analysis has emerged as an effective approach for malware detection. However, extracting actionable intelligence from noisy execution logs while maintaining model interpretability remains a challenge. To address this, we propose a multi-layered framework that connects raw behavioral signals with standardized threat intelligence. The architecture utilizes a Log2 noise filtering mechanism to distill execution traces into API segments, which are then mapped into semantic representation spaces using CodeBERT. By leveraging HDBSCAN for density-based clustering alongside PrefixSpan, the system extracts sequential patterns representing core malicious activities. For classification, a soft-voting Ensemble model (Random Forest and XGBoost) is employed, achieving an accuracy of 99.40%, with feature-level interpretability provided by SHAP values. Furthermore, we integrate a Retrieval-Augmented Generation (RAG)-based LLM agent to interpret technical signals and assist in mapping behaviors to the MITRE ATT&CK framework. Experimental results on Windows malware datasets demonstrate that the proposed framework can effectively identify obfuscated threats and automatically generate structured digital forensics reports to support Incident Response and DFIR workflows.