Testing the Robustness of IoT Messaging Middleware: A Case Study Using Rules and Large Language Models

Abstract

The Internet of Things (IoT) allows the inter-connection of objects, equipped with specialized hardware and software, in complex networked systems designed to fulfill business goals. IoT is frequently supported by asynchronous messaging middleware, which must operate correctly even in the presence of invalid or malicious data. While prior research has examined the robustness of asynchronous applications, the assessment of the corresponding middleware is relatively scarce with known studies pointing it out as being mostly robust, at least when using traditional techniques. In this paper, we assess the robustness of messaging middleware (i.e., ActiveMQ Classic 6.1.5, RabbitMQ 4.0.5) designed for IoT systems and that implement typical IoT communication protocols (i.e., MQTT 3.1.1, STOMP 1.2, AMQP 1.0) in the following two ways: i) we generate test cases using traditional robustness testing techniques; and ii) we use three off-the-shelf large language models (i.e., gemma3_12b, qwen2.5-coder_32b, mistral-small3.2) to generate test cases by using three distinct prompts. Results show that message brokers are highly robust, whereas client libraries are the weakest link, failing more often with invalid inputs, typically by forwarding known invalid messages to lower layers. The models increased fault diversity, especially for security-oriented faults, but did not consistently outperform rule-based testing, matching it at an equal test budget and surpassing it in 3 of 6 scenarios only when combining multiple roles, i.e., using a larger cumulative number of tests.