Enhancing IDS Robustness through Adaptive Statistical Feature Selection with Kneedle-Driven Thresholding

Abstract

The rapid advancement of Internet of Things (IoT) ecosystems demands an Intrusion Detection System (IDS) architecture evolution to maintain a secure network. Statistical filters such as Chi-Square, ANOVA, and Pearson Correlation have worked well, but they usually rely on fixed thresholds, which do not work well across different environments. This study proposes a feature selection pipeline that utilizes Kneedle to determine optimal feature subsets. This method is evaluated using the NSL-KDD and IoTID20 datasets, a Decision Tree as the main classifier, and compared against a fixed threshold baseline of K=15. Results show better performance than the baseline, achieving 98.92% accuracy and 1.01% FPR on NSL-KDD using Pearson Correlation compared to 96.93% and 2.58% under fixed thresholding. Both methods achieved an accuracy range of 99.93-99.94% on IoTID20, with the adaptive method achieving a lower FPR. Cross-dataset evaluation confirms that adaptive thresholding drops the standard deviation of performance metrics by up to 75%. Additionally, the inference latency per sample reduced from 0.00009 ms to 0.00008 ms, while the memory footprint was almost undetectable. These findings show the potential of Kneedle-driven thresholding as an effective, adaptive, and lightweight alternative to static feature selection for IoT IDS.