A Distributed Mixture-of-Experts Framework for Cross-Dataset Intrusion Detection Using a Universal Flow Representation

Abstract

Intrusion Detection Systems (IDS) trained on individual benchmark datasets often achieve high accuracy but fail to generalize across heterogeneous network environments due to domain shift. To address this challenge, this paper proposes U-Flow MoE-IDS, a distributed Mixture-of-Experts (MoE) framework for cross-dataset intrusion detection. The framework introduces a Universal Flow (U-Flow) feature schema that aligns heterogeneous traffic attributes from NSL-KDD, CICIDS2017, and Bot-IoT into a unified representation. Dataset-specific expert models are trained using Apache Spark, and a leakage-free stacking-based logistic regression meta-learner aggregates expert predictions for adaptive decision fusion. Experimental results demonstrate strong intra-dataset performance, achieving Macro-F1 scores above 0.97. However, the Leave-One-Dataset-Out evaluation reveals performance degradation, with Macro-F1 scores of 0.902, 0.531, and 0.477 across different holdouts, highlighting persistent domain shift challenges. Experimental comparison with a global Random Forest baseline demonstrates the effectiveness of the proposed expertbased framework in cross-dataset evaluation. The results indicate that expert specialization provides greater robustness and generalization than conventional pooled training when dealing with heterogeneous datasets. The proposed framework provides a scalable distributed IDS architecture and enables systematic evaluation of cross-dataset generalization for heterogeneous network environments.