Large Language Models for phishing URL detection: A comparative study of LLaMA-3, GEMMA-7B, and traditional Machine Learning approaches

Abstract

Phishing attacks remain a major cybersecurity threat, causing financial loss, data breaches, and reputational damage to individuals and organizations. Conventional detection methods, such as rule-based filters and blacklist-based systems, often fail to identify newly generated or highly obfuscated phishing URLs. Recent advances in artificial intelligence, particularly large language models (LLMs), provide new opportunities to detect complex structural and contextual patterns in malicious web addresses. This study evaluates the effectiveness of transformer-based LLMs for phishing URL detection by comparing LLaMA-3 and GEMMA-7B with traditional machine learning models, including Support Vector Machines, Random Forest, and XGBoost. The models were fine-tuned and tested on a publicly available dataset containing 247,950 labeled URLs, each described by 42 structural and lexical features. To support transformer-based classification, structured URL attributes were converted into textual representations, enabling the models to exploit contextual embeddings. Experimental results showed that LLMs outperformed conventional machine learning baselines. GEMMA-7B achieved the best performance, with an accuracy of approximately 94.8%, followed by LLaMA-3 at 94%. These findings suggest that transformer-based models can effectively identify deceptive domain structures, abnormal URL patterns, and obfuscation techniques. Overall, the proposed LLM-based framework offers a promising approach for improving phishing detection and strengthening modern cybersecurity defenses.