Detecting Malicious Nodes Using Game Theory and Reinforcement Learning in Software-Defined Networks

Abstract

Software-Defined Networking (SDN) has become a widely adopted networking paradigm due to its centralized management and enhanced flexibility. However, the centralized architecture of SDN makes it susceptible to security threats such as malicious nodes and botnet attacks, which can negatively impact network performance, availability, and reliability. Conventional intrusion detection techniques often depend on predefined signatures and static rules, limiting their ability to detect sophisticated and emerging cyber threats. To overcome these limitations, this study presents a Mafia Game-Based Malicious Node Detection Framework that utilizes role-based modeling, where network entities are represented as roles including Godfather, Mafia, Detective, Doctor, and Townie for effective behavioral analysis and trust assessment. The proposed framework integrates belief-based evaluation mechanisms with Reinforcement Learning (RL) to detect, prioritize, and classify malicious nodes within the SDN environment. By continuously learning from network interactions and previous outcomes, the RL agent enhances the adaptability and accuracy of the detection process. The performance of the framework is assessed using standard evaluation metrics such as Accuracy, Precision, Recall, F1-Score, True Positive Rate (TPR), and True Negative Rate (TNR). Experimental findings indicate that the proposed method effectively identifies malicious activities and strengthens overall network security. The framework offers an intelligent, adaptive, and scalable approach for safeguarding modern Software-Defined Networks.