A Dual-Tier Framework for Intrusion Detection and Mitigation in 5G Open-RAN

Abstract

Open-RAN is transforming 5G and Beyond-5G networks by promoting openness, flexibility, and interoperability across network components. However, this architectural shift introduces new security challenges due to expanded attack surfaces, distributed control mechanisms, and exposed interfaces. Traditional Intrusion Detection Systems (IDS) struggle to address the dynamic threat landscape of O-RAN environments. To address this challenge, we propose ORANGuard, an Artificial Intelligence-driven dual-tier intrusion detection and mitigation framework for O-RAN environments. ORANGuard comprises two core modules: Real-Time Guard (RT-Guard), a real-time threat detection model deployed in the Near-Real-Time RAN Intelligent Controller (Near-RT RIC), and Non-Real-Time Guard (NRT-Guard), a periodic anomaly analysis model operating within the Non-Real-Time RIC, together enabling multi-level threat detection. The framework is built on an end-to-end 5G O-RAN testbed using srsRAN and Open5GS, capturing E2 measurements under both benign and attack conditions. Custom scripts simulated anomalous scenarios, including DDoS, producing a dataset of 17,052 E2 measurement records across a testbed of 4 gNodeBs and 12 UEs. RT-Guard, based on Random Forest, achieved 96.7% accuracy, while NRT-Guard, using an Autoencoder, reached 95.6%. The hierarchical dual-tier decision rule achieved 99.5% accuracy on the held-out test set. RT-Guard required a median detection latency of 5.2 ms and a median detect-and-enforce latency of 17.9 ms, whereas NRT-Guard required 283.6 ms after closure of its one-second analysis window. These results demonstrate the complementary low-latency and anomaly-analysis capabilities of ORANGuard for O-RAN security.