An advanced framework for malware detection using adversarial learning

Abstract

Malware is a type of software designed to intentionally disrupt or gain unauthorized access to a computer system. As technology progresses, new variants of malware continue to emerge. In the domain of image-based Windows PE malware family classification, modern deep learning models are highly susceptible to adversarial attacks, leading to biased learning between clean and adversarial samples, suboptimal retraining, and reduced capability to capture family-specific discriminative features. Existing attack generation methods rely heavily on cybersecurity expertise, making them complex and difficult to scale. Additionally, retraining often overfits adversarial samples or loses sensitivity to clean data and weakening defenses. To address these issues, this paper proposes a systematic framework to enhance adversarial robustness, validated on the Malimg dataset. Firstly, malware binaries are converted into grayscale images, enabling image-based deep learning analysis. Secondly, a multi-stage oversampling strategy is applied before and after generating adversarial samples to reduce bias and improve data diversity. Thirdly, a novel feature mask is introduced, combining Convolutional Neural Network (CNN) derived gradients, structural masks from PE parsers, and unused null byte positions. This mask selectively alters non-critical regions, preserving malware functionality while reducing adversarial noise. Finally, to address long-range dependencies, adversarial samples are tested against Vision Transformers (ViTs) in a black-box environment, ensuring transferability and improving detection across various models. Detailed analysis of the results reveals the achievement of the 81% evasion rate with the average L0 norm 0.284%. However, retraining the model increases the accuracy from 16 to 97%. These results are mirrored on the MalEVis dataset, where a 94.40% F1-score is maintained and accuracy is restored to 90.79% after byte-level perturbations, validating the framework’s scalability. Compared to existing methods, the proposed model supersedes previous approaches by 13.15% in terms of the evasion rate and 37.34% in terms of retraining success.