Regime-Aware Deployment Validity of Flow-Based IDS

Abstract

Flow-based intrusion detection studies often report strong benchmark scores, but their relevance to future traffic remains uncertain. This paper treats IDS behavior as regime-dependent and proposes an evaluation protocol under temporal shift, family novelty, and combined shift. Flow-only, context-only, selected-context, and full-context XGBoost variants are evaluated on CICIDS2017 and UNSW-NB15 using regime composition, ablations, bootstrap confidence intervals, five-seed sensitivity for corrected family holdouts, diagnostic score distributions, precision–recall analyses, and approximate runtime. CICIDS2017 random-split performance is near perfect, whereas future and family-novel regimes differ sharply. Selected graph-derived context improves Thursday web OOD and combined shift, while full context can reduce transfer. Friday botnet distribution shift and the botnet-family holdout remain severe failures, with zero F1 and recall for several feature sets. In contrast, the record-disjoint UNSW-NB15 Generic/Exploits holdout remains strong for the flow-only model, showing that OOD difficulty is regime-dependent. The results demonstrate that high random-split performance does not establish deployment readiness; temporal evaluation can also mislead when attack-family structure is not separated, and graph-context transfer varies by feature set and operating regime.