Robust Statistical Anomaly Detection for Interpretable Intrusion Monitoring in Complex Network Systems

Abstract

Anomaly detection is essential for identifying abnormal observations that may signal operational errors, system failures, fraud, network misuse, or security threats in complex data systems. This study develops and evaluates a robust statistical framework for interpretable anomaly detection using the Canadian Institute for Cybersecurity Intrusion Detection System 2017 (CICIDS2017) network intrusion dataset as an empirical case. The framework combines descriptive profiling, data-quality auditing, benign-baseline calibration, robust location and scale estimation, interquartile range (IQR) boundaries, median absolute deviation (MAD) scoring, empirical probability-tail thresholds, and feature-count aggregation. The empirical dataset contains 2,830,743 observations distributed across eight flow-based files, including 2,273,097 benign records and 557,646 attack or anomaly records. The Monday benign-only traffic was used to calibrate the normal statistical baseline, while 2,300,825 observations from the remaining days were retained for holdout evaluation. Sixteen interpretable flow variables were selected for the statistical monitoring experiment after cleaning missing values, infinite values, and inconsistent labels. The most conservative probability-tail rule reached 84.80% accuracy, 87.30% precision, 43.63% recall, an F1-score of 0.5818, and a false-alarm rate (FAR) of 2.03%. The findings show that robust threshold monitoring is particularly useful when the main operational objective is high-confidence early warning with low false alarms rather than maximum attack recall. The study contributes a transparent, reproducible, and statistically grounded anomaly-detection procedure that can support cybersecurity monitoring and can be adapted to other risk-sensitive complex data systems.