Network Traffic Anomaly Detection for Server Security Using a Hybrid CNN–BiLSTM and LightGBM Optimized by Whale Optimization Algorithm

Abstract

The rapid growth of server-based digital services (e.g., web services, databases, cloud platforms, and IoT gateways) expands the attack surface, while threats such as DDoS, brute-force attempts, port scanning, web attacks, bot activity, and infiltration are increasingly difficult to distinguish from legitimate traffic. This study proposes a hybrid anomaly detection pipeline that combines deep learning for representation learning with gradient-boosted decision trees for multiclass classification on structured traffic features. The pipeline applies Min–Max scaling to harmonize feature ranges and SMOTE to mitigate class imbalance, then uses a CNN–BiLSTM network to capture local patterns and temporal dependencies, followed by LightGBM as the final classifier. To enhance LightGBM performance, its hyperparameters are optimized using the Whale Optimization Algorithm (WOA). We evaluate the approach using accuracy, precision, recall, and F1-score in a multiclass setting. Experimental results show that the CNN–BiLSTM + LightGBM baseline achieves 0.92 accuracy with strong performance on most attack categories, while rare and ambiguous classes (notably Infiltration and Web Attack) remain challenging. With WOA-based optimization, overall accuracy increases to 0.93 and the macro-average F1-score improves from 0.88 to 0.89, with clearer gains for several time-dependent attack classes. While the proposed framework is designed with practical deployment in mind by leveraging a tree-based classifier on learned features, this study does not report system-level efficiency measurements (e.g., training time, inference latency, or memory footprint); these will be addressed in future work to substantiate computational feasibility in real server environments.