State-Aware Hybrid Concrete-Symbolic Analysis for Automated Vulnerability Discovery in Iot Communication Protocols

Abstract

Internet of Things (IoT) devices heavily rely on lightweight communication protocols such as MQTT, CoAP, and 6LoWPAN. To minimize memory footprint and computational overhead, these protocols are often implemented using custom C/C++ code, making them highly susceptible to memory corruption vulnerabilities including buffer overflows and integer underflows. Traditional automated testing techniques, such as coverage-guided fuzzing and standard symbolic execution, struggle to effectively explore the deep state spaces of these protocols due to the complex, stateful nature of network handshakes and the path explosion problem. In this paper, we propose ProtoSym, a novel State-Aware Hybrid Concrete-Symbolic Engine designed specifically for IoT protocol vulnerability discovery. Our approach synergistically combines directed concrete execution to navigate complex network state machines with selective symbolic execution to rigorously solve constraints within packet payloads, effectively finding deep-seated security flaws in IoT gateway firmware. We introduce a formal theoretical framework that models protocol states and transitions, enabling precise constraint formulation, and present a directed path exploration strategy that mitigates path explosion while maximizing state coverage. Experimental evaluation on widely-used IoT protocol implementations demonstrates that ProtoSym achieves up to 46.7% higher statespace coverage and discovers $3.7 \times$ more unique vulnerabilities compared to state-of-the-art baselines.