A Family-Aware Hierarchical XGBoost Framework for Efficient IoT Intrusion Detection

Abstract

: Machine learning-based intrusion detection for Internet of Things (IoT) networks remains difficult because modern traffic is highly imbalanced and attack behaviors are heterogeneous. Evaluation pipelines can also overestimate performance when preprocessing is performed before train-test separation. We propose a family-aware hierarchical intrusion detection framework for attack-family prediction. The proposed approach first separates normal and attack traffic, then routes attack samples into empirically defined majority and minority attack-family branches, and finally performs branch-specific family classification. Within each cross-validation fold, training-label counts define the majority/minority routing branches, while scaling, weighting, model fitting, stage diagnostics, and metric computation remain fold-local. The final implementation uses XGBoost as the base learner in the hierarchical stages and compares it with flat LightGBM, XGBoost, Random Forest, Extra Trees, and stacking baselines under matched folds and metrics. On the CICIoT2023 30% stratified development split, the proposed approach achieved a Macro-F1 of 0.8380 and Weighted-F1 of 0.9940, performing close to the best flat Random Forest baseline while improving weak rare-family F1 scores for BruteForce and Web. On Edge-IIoTset, where the full processed dataset is used, the proposed approach achieved a Macro-F1 of 0.9456 and Weighted-F1 of 0.9494, outperforming all individual flat baselines and approaching the flat stacking ensemble. The hierarchy had lower inference time than the evaluated flat baselines under the workstation protocol.