From KNN to Ensemble Learning: Advancing Real-Time Network Intrusion Detection with Asymmetric SMOTE-ENN Balancing on the CSE-CIC-IDS2018 Benchmark

Abstract

The increasing occurrence and sophistication of attacks on current enterprise-grade computer networks require the development of IDSs with real-time, per-flow traffic categorization capability while ensuring a low rate of false positives. Most traditional architectures based on machine learning models are typically plagued with the “False Positive Trap” - a spatial boundary deviation issue caused by oversampling techniques, such as the popular SMOTE, which expand the size of extreme minority classes (such as the Web Attack class with only 8 natural occurrences) to be equivalent to the number of instances of the majority benign category, leading to the misclassification of legitimate flows as attacks through distance-based classifiers. This paper aims to solve this primary issue with the introduction of our new Asymmetric SMOTE-ENN data pipeline augmented with a champion selection of Optimized K-Nearest Neighbours classifier tested against the CSE-CIC-IDS2018 dataset, a comprehensive collection of attack samples collected from a simulation of a large-scale enterprise-grade network environment with seven types of attacks. The proposed pipeline adopts an asymmetric class limit where there will be a maximum of 250,000 benign records as a realistic statistical prior, limiting attack classes to 75,000 records, and ensuring that the extreme minority classes have at least 10,000 synthetic samples generated after applying Edited Nearest Neighbours (ENN) to filter out noisy overlaps between artificial and natural samples. The selected attributes for the Gini impurity-based Random Forest feature selector are reduced from the 78 attributes of the raw CICFlowMeter dataset to the top 20, with a 74.4% dimensional reduction efficiency while maintaining classification accuracy. Our champion optimized KNN classifier achieves 95.7% overall accuracy, 1.9% false positive rate in the Benign class, and 3.2 ms inference time per flow, allowing us to implement real-time mitigation capabilities using a Django-Channels WebSocket pipeline, yielding an end-toend API response of 6.8 ms. The overfitting analysis using 5-Fold Cross-Validation shows excellent generalization with a Train/Test accuracy difference of only 0.11%.