Beyond Aggregate Accuracy: Attack-Family Reliability of Machine-Learning Intrusion Detection on TON_IoT

Abstract

Aggregate intrusion-detection scores can conceal substantial differences in how reliably individual attack families are detected. This study evaluates attack-family reliability of three conventional machine-learning classifiers on the TON_IoT network dataset under a leakage-controlled protocol. The checksum-verified CSV contained 211,043 rows and 44 columns; 20,569 exact whole-row duplicates were removed. After excluding the binary and multiclass targets and identifier/high-leakage-risk fields, 31 predictors were retained. A representation-support audit was then performed because repeated model-visible feature patterns can limit the interpretability of family-level grouped generalization. An eligibility rule established after this diagnostic audit and before the final out-of-fold (OOF) model comparison retained seven classes—normal, backdoor, DDoS, DoS, injection, MITM, and password—yielding 140,602 observations; ransomware, scanning, and XSS were reported separately because of high retained-representation concentration. Five-fold StratifiedGroupKFold evaluation grouped identical retained-feature representations and achieved zero train-test signature overlap in every fold. HistGradientBoosting (HGB) achieved the strongest aggregate performance (accuracy 0.90397, macro-F1 0.86766, MCC 0.88268, balanced accuracy 0.87113) and the strongest worst-family recall (0.69124). However, superiority was family-dependent: HGB raised DoS recall to 0.6912 versus approximately 0.29 for Random Forest (RF) and Decision Tree (DT), while RF/DT materially outperformed HGB for backdoor and MITM. The results show that aggregate superiority does not imply uniform attack-family superiority and that family-level reliability should be interpreted together with representation support and leakage-controlled evaluation.