Skip to content
Open access

A Comparative Analysis of Machine Learning Robustness Against Concept Drift in Phishing URL Detection

Oct 2026 · Bulletin of Network Engineer and Informatics · Vol 4, pp. 547-560 · 0 citations

TL;DR

XGBoost demonstrated the highest relative robustness in maintaining detection performance throughout the testing period, indicating that XGBoost demonstrated the highest relative robustness in maintaining detection performance throughout the testing period.

Abstract

Machine learning-based phishing URL detection may experience performance degradation when the characteristics of incoming data change over time, a condition associated with concept drift. This study aims to evaluate and compare the robustness of XGBoost, Random Forest, and LightGBM in detecting phishing URLs using lexical features under changing data conditions. The dataset was collected from PhishTank and Tranco and temporally partitioned into training data from 2023–2024 and testing data from 2025. Each URL was represented using 36 lexical features, and model performance was evaluated across four quarterly periods (Q1–Q4). Robustness was analyzed using ADWIN drift detection, the Friedman test followed by Bonferroni-Dunn post-hoc analysis, and Area Under the Time (AUT) based on F1-Score. The models achieved high validation performance, with F1-Scores ranging from 0.9855 to 0.9868, with XGBoost achieving the highest value (0.9868), followed by LightGBM (0.9864) and Random Forest (0.9855). During the temporal evaluation, all models experienced a declining F1-Score trend from Q1 to Q4. This degradation, which exhibited a statistically significant difference between Q1 and Q4, was primarily driven by a sharp decline in Recall rather than Precision. ADWIN detected performance changes at different points during the testing period. XGBoost achieved the highest AUT of 0.9721, followed by LightGBM (0.9706) and Random Forest (0.9694). These findings indicate that XGBoost demonstrated the highest relative robustness in maintaining detection performance throughout the testing period.

Read PDF

We use cookies to run the site and, with your consent, for analytics and to show ads. See our Cookie Policy.