A hybrid eBPF/ML intrusion detection system with two-model heterogeneous ensemble learning and real-time gateway-mode network blocking

Abstract

Network intrusion detection systems (IDS) face a persistent trade-off between detection accuracy and processing latency. Conventional ML-based IDS implementations operate entirely in user space, introducing per-packet overhead from kernel-to-user-space context switches and memory copies which becomes the dominant cost at moderate-to-high traffic rates. We present a hybrid architecture that pushes 23-dimensional flow-feature extraction into the Linux kernel using an eXpress Data Path (XDP) eBPF program, and combines it with a heterogeneous, calibrated two-model ensemble in user space. The kernel program parses Ethernet/IP/TCP/UDP headers, maintains bidirectional per-flow state in an LRU hash map, and submits cumulative features to user space through a 4 MB lock-free ring buffer. A sliding-window detector then feeds two complementary supervised models—a Deep Neural Network (DNN) and an isotonic-calibrated Random Forest (RF)—whose probabilities are blended through validation-selected weighted voting under a confidence gate and a consecutive-attack confirmation rule. We further describe a gateway-mode deployment that turns the IDS host into the network gateway for protected devices, enabling true device-level isolation through paired eBPF XDP_DROP and iptables-FORWARD blocking. To address methodological concerns of an earlier draft, the system is re-evaluated under a strict per-day, per-attack-type chronological train/validation/test split on CIC-IDS2018, with held-out hyperparameter selection (including a one-dimensional grid search over wdnn\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{\text {dnn}}$$\end{document} with wrf=1-wdnn\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$w_{\text {rf}}=1-w_{\text {dnn}}$$\end{document}), a component ablation matrix, τ/k/\documentclass[12pt]{minimal} \usepackage{amsmath} \usepackage{wasysym} \usepackage{amsfonts} \usepackage{amssymb} \usepackage{amsbsy} \usepackage{mathrsfs} \usepackage{upgreek} \setlength{\oddsidemargin}{-69pt} \begin{document}$$\tau /k/$$\end{document}window sensitivity sweeps, a per-attack false-negative breakdown, 1 000-resample bootstrap 95% confidence intervals, and a cross-dataset evaluation on CIC-IDS2017 (same models, same pipeline, with explicit transfer experiments in both directions). On the CIC-IDS2018 chronological test fold (mean across three seeds) the validation-tuned DNN + RF ensemble achieves accuracy 0.9567, precision 0.9242, recall 0.8920, F1 0.9079, FPR 0.0230 and ROC-AUC 0.9707 (95% bootstrap CIs on F1: [0.9046, 0.9124]); the in-distribution F1 is statistically indistinguishable from a pure-RF model, but the cross-dataset transfer to CIC-IDS2017 illustrates the value of the heterogeneous blend. End-to-end kernel-to-decision latency is 1.4 ms median on a single Intel i7-12700H core. A web dashboard provides per-model vote visualisation, blocking controls, and operator telemetry. All code, models, hyperparameters, random seeds, and the full container recipe required to reproduce every reported number are released alongside the manuscript.