Multi-Level Distributional Entropy from Flow Summary Statistics for Explainable Network Intrusion Detection

Abstract

Machine learning network intrusion detection systems (IDS) operate on aggregate flow statistics that discard the distributional structure of traffic, and although information-theoretic measures capture that structure, established entropy estimators require raw packet sequences that pre-aggregated flow datasets do not contain. No prior method derives entropy from the summary statistics those records already hold. We introduce Multi-Level Distributional Entropy (MDE), which computes interpretable information-theoretic features analytically from flow-level summary statistics at three levels, within-flow Gaussian differential entropy, cross-directional Jensen-Shannon divergence (JSD), and Transmission Control Protocol (TCP) flag-incidence Shannon entropy, with closed-form properties and no raw packet access; only imputation medians and score bounds are fitted on the training split. We pair the features with a leakage-free, fold-local evaluation protocol that reports the full operational metric suite across cross-validation, temporal, pseudo-live, cross-dataset, and unseen-attack-family settings, on four benchmarks (NSL-KDD, CICIDS-2017, CICIDS-2018, UNSW-NB15) with tree-ensemble classifiers and SHAP. The protocol exposes failure modes that aggregate weighted F1 conceals: on CICIDS-2018 an F1 of 0.73 hides a detection rate (DR) of 0.44, on held-out attack families F1 exceeds 0.998 while DR falls to zero, and a 703K-flow pseudo-live replay reveals a threshold-ranking divergence in which score ranking is largely preserved (area under the ROC curve, AUC, 0.84 to 0.86) while fixed-threshold detection collapses (DR 0.08). The entropy features match conventional features within 0.1 percentage points of F1 and receive reproducible SHAP attributions (Spearman 0.84 to 0.94), so their contribution is a grounded, interpretable representation and an evaluation methodology rather than an accuracy gain.