Generalizing across Networks: Evaluating Model Transferability for Intrusion Detection

Abstract

: As cyber threats continue to evolve, attackers improve their techniques to optimize the effectiveness of their malicious activities. In response, there has been a significant increase in the implementation of Machine Learning (ML) and Deep Learning (DL) in Network Intrusion Detection Systems (NIDS). Choosing high-quality, representative datasets is a critical step in improving detection accuracy; however, many datasets have been available for a long time, some for over two decades, raising questions about the ability of ML and DL models to generalize across different and possibly outdated network scenarios. This study evaluates the generalization capability of models such as LGBM, RF, XGB, and LSTM, particularly in identifying previously unseen attacks. A significant challenge addressed is the inconsistency introduced by different flow exporters, which compute features differently, affecting model reliability. To mitigate this, HERA is used to rebuild datasets directly from original PCAP files. Additionally, we investigated the impact of feature selection on generalization and examined how performance changes when combining different datasets. When evaluated on alternative datasets, however, their behavior diverges, and in some cases, similar traffic patterns allow for reasonable generalization, while in others, the uniqueness of the dataset leads to significant performance drops.