AirVScan: Are LLMs Capable of Detecting and Reasoning about Wireless Attacks?

Abstract

The inception of large language models (LLMs) is expected to revolutionize the software security analysis pipeline. However, is it possible to use state-of-the-art LLMs for detecting and reasoning about the attack vector of real-world wireless attacks? In this paper, we propose, design and evaluate AirVScan– a generic workflow to study the current capabilities of LLMs in light of analyzing wireless attacks. AirVScan takes a packet sequence, possibly containing an attack, as input and as an output, it produces the attack vector (e.g., abnormal packet field) with reasoning if the attack is detected. AirVScan differs from works that investigate LLMs to understand the security of source code, as our approach merely takes the packet sequence as an input. We implement the workflow of AirVScan and evaluate it with state-of-the-art LLMs (o3, grok-4, gemini-2. 5-pro) against 40 benign test cases and 128 attack test cases. Among them, 80 attack test cases exploit real-world implementation vulnerabilities in two widely used wireless protocols (Bluetooth and 5G NR). Our evaluation reveals that AirVScan correctly reasons about the root cause of wireless attacks with a rate of 60.7% for Bluetooth but with only 7.7% for 5G NR, highlighting the limitations of current LLMs to reason about attacks on complex protocol implementations. We also evaluated AirVScan to study the effectiveness of LLMs for an unfamiliar protocol that we created by obfuscating Modbus. For such an unfamiliar protocol, AirVScan was mostly ineffective at reasoning about complex, synthesized attacks. However, the inclusion of context improved the attack vector identification to over 82.1%. Finally, we compare AirVScan with AirBugCatcher-a competitive and customized tool for analyzing attack vector of wireless attacks. Our results shed light on the LLMs' current capabilities and required future research for detecting wireless attacks.