A Systematic Multi-Paradigm Evaluation Framework for Network Intrusion Detection in Fog-IoT Environments: Deep Learning, Transformer, and Ensemble Methods Across Deployment Tiers

Abstract

This paper presents the Adaptive Confidence-Gated Ensemble (ACGE) framework for Network Intrusion Detection Systems (NIDSs) in resource-heterogeneous fog-IoT deployments. No single algorithmic paradigm simultaneously achieves high minority-class detection accuracy, sub-millisecond inference latency, and generalization across heterogeneous IoT traffic at resource-constrained fog-IoT nodes. To address this challenge, ACGE integrates a Sequence Transformer Encoder with LightGBM via a learned and sample-adaptive gating network trained on inter-model error complementarity, formally specified within a three-mode tiered deployment architecture. Both models execute in parallel for every sample; the gating network assigns transformer-dominant weight (α ≈ 1) to 18% of samples where attention-based processing yields the greatest architectural advantage. Under true parallel inference, ACGE latency is 1.45 ms (bounded by the transformer); the measured wall-clock latency under CPU-GPU pipeline parallelism is 1.52 ms. The framework is evaluated under 5-fold stratified cross-validation with McNemar’s significance testing on IoTID20 (primary IoT benchmark) and NSL-KDD (reproducibility anchor) and validated through zero-shot transfer on CICIoT2023 and Edge-IIoTset. Notably, the Sequence Transformer Encoder and ACGE process L = 5 consecutive flow records per decision, providing five times the information available to single-flow baseline models; approximately 57% of the transformer’s performance advantage is attributable to architectural design rather than extended input access. On IoTID20 multiclass classification, ACGE is the only evaluated configuration yielding a negative accuracy-to-macro-F1 gap (−0.51 pp), reaching 99.12 ± 0.03% accuracy and 98.67 ± 0.08% balanced accuracy, demonstrating genuine minority-class improvement. LightGBM achieves the optimal fog-tier profile (0.97 ms latency, concept drift sensitivity of −0.53 pp over three months). On Edge-IIoTset zero-shot transfer, ACGE achieves 93.89% binary accuracy and 85.78% multiclass macro-F1. Ransomware detection (12.34–66.34% F1 across all models) is identified as the primary capability gap; targeted data collection, few-shot adaptation, and federated learning are recommended as the most critical future directions. All inter-paradigm performance differences are reported as the mean ± SD and verified by McNemar’s test with Bonferroni correction.