Development of a Real-Time Flow-based Intrusion Detection and Security Management System using a Two-Stage Ensemble Model

Abstract

The increasing complexity of modern network environments has heightened the need for practical and scalable intrusion detection solutions. This study presents the development of a real-time, flow-based Intrusion Detection and Security Management System (IDSMS) that integrates live traffic monitoring, machine learning-based intrusion analysis, centralized data storage, and web-based visualization. The system captures network traffic from a MikroTik-based gateway using a passive flow sniffer deployed on a Kali Linux monitoring node. Packets are aggregated into bidirectional flows and transformed into statistical feature vectors, which are analyzed by a machine learning inference service through an application programming interface. Detection results, including risk level and attack classification, are stored in a centralized database and presented through an interactive dashboard. The intrusion detection engine employs a two-stage ensemble approach: Stage A performs binary classification (benign vs. malicious) using a calibrated stacking ensemble of Random Forest, Gradient Boosting, and XGBoost classifiers with Logistic Regression as meta-classifier, achieving 99.92% accuracy and 0.99987 ROC-AUC. Stage B performs multi-class classification across 14 attack categories using a Random Forest classifier, achieving 99.83% accuracy with per-class F1-scores exceeding 99.6%. Development-stage testing using simulated attack traffic demonstrates the system’s ability to capture, analyze, and visualize network threats in real time, with average inference latency of 12.4 ms per flow and processing throughput of 3,842 flows per minute. The proposed system provides a practical foundation for deployment in shared-access networks, small enterprises, and IoT-enabled environments.