The 99 Percent Accuracy Illusion: A Base-Rate-Aware and Cross-Dataset Re-Evaluation of Machine-Learning Intrusion Detection

Abstract

Machine-learning (ML) intrusion-detection systems are routinely reported to achieve 98-99% accuracy on standard benchmarks, widely read as evidence that the problem is solved. We challenge that reading through a controlled, leakage-safe, reproducible re-evaluation of four representative classifiers logistic regression, random forest, XGBoost and a multilayer perceptron - on the NSL-KDD benchmark and, for cross-network generalization, on the NetFlow-v2 datasets NF-UNSW-NB15 and NF-ToN-IoT. Four findings follow. First, the high accuracy is an artifact of same-distribution validation: five-fold cross-validation reproduces 97.4-99.9% accuracy, yet the identical pipelines attain only 75.3-79.5% on the held-out, distribution-shifted test partition, an optimism gap of 20-22 points. Second, this reduced accuracy is still operationally misleading: projecting each model's measured true- and false-positive rates onto traffic in which intrusions are 0.1% of flows, the precision of the best model collapses from 0.96 to 0.023, so over 97% of alarms are false and even that model raises more than $\text{2 6, 0 0 0}$ false alerts per million flows. Third, detection is highly uneven: recall on rare remote-to-local attacks falls to 1%, and recall on attack types unseen during training (25-53%) is far below that on known types (71-78%). Fourth, in a true cross-network experiment the Matthews correlation coefficient of the strongest models collapses from 0.96 to approximately zero, and in places negative. These results expose a systemic evaluation gap rather than an algorithmic deficiency, and we propose a base-rate-aware, distribution-shift-aware evaluation protocol as a prerequisite for credible ML-IDS research.