Advanced detection technique for memory-resilient cyber attacks
Abstract
One of the most significant strategies attackers to evade a system is to force a legitimate application to misbehave, which may be interpreted as abusing its native features. Since most companies trust and use Microsoft Office (MS), adversaries consider it a promising target, i.e., an attack vector. This paper addresses the critical and evolving threat of in-memory payload (shellcode) execution, particularly in ubiquitous MS Word document-processing applications. Conventional defense mechanisms, such as Address Space Layout Randomization (ASLR), Data Execution Prevention (DEP), and Control-Flow Integrity (CFI), have proven insufficient against robust malware attacks (i.e., malicious shellcode built with security in mind) and code-reuse attacks. This research reviews contemporary payload injection vectors, such as malicious macros and Object Linking and Embedding (OLE) vulnerabilities. It examines the limitations of current detection paradigms, including Endpoint Detection and Response (EDR) and memory forensics. Motivated by these challenges, a novel, multifaceted defense framework, termed Flexible & Sturdy Memory Integrity Guardian (FSMIGuard), is proposed. FSMIGuard integrates advanced API call sequences for anomaly detection, dynamic memory region profiling using machine learning (ML), and refined entropy-based rarity (anomaly) scoring. Its architecture leverages hardware-assisted monitoring for minimal overhead and advanced improvement. Through meticulous empirical evaluations on a testbed comprising \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$N = 625$$\end{document} security-relevant nodes, this research also examines the efficacy of detection frameworks against complex evasion vectors, such as Documents-inside-Documents (DiD) techniques. The empirical results demonstrate robust detection capabilities, yielding an aggregated F1-score of \documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$93.20\%$$\end{document} for cross-layer anomaly detection, with detailed evaluations encompassing precision, recall, and accuracy across multi-layered payload architectures. These findings highlight critical avenues for strengthening defensive architectures and mitigating structural vulnerabilities in enterprise environments.