A comparative evaluation of large language models for detecting SQL injection vulnerabilities in web applications

Abstract

Structured Query Language (SQL) injection is considered to be one of the most intractable and harmful threats to the security of web applications, as SQL injection is a technique that allows hackers to manipulate queries and steal confidential information. Even though traditional methods of defense have been used, emerging attack techniques continue to circumvent them, and it requires stronger mechanisms of detection. As the concept of Large Language Models (LLMs) has become prominent in the field of cybersecurity, the idea of assessing their capability to detect and address such threats in practical contexts is gaining more and more popularity. This article provides a comparative evaluation of five pre-trained LLMs on the capability of detecting SQL injection attacks. To induce more realistic conditions, a Flask-based web application that linked to an SQLite database was created, which was able to support both benign queries and SQL injection payloads using Boolean-based, Union-based, and Error-based methods. These models were put to the test in the zero-shot settings, and their performance was evaluated by accuracy, precision, recall, and F1-score. Findings indicate that Mixtral-8x7B-Instruct achieved the best results, with an average F1-score of 87.52% and an accuracy of 86.67%. The second-best performance was achieved by LLaMA-3-70B-Instruct, which demonstrated consistently high recall across all attack categories. Deep Seek and CodeLLaMA both performed well in some form of attacks, but not overall. In contrast, Qwen2.5-Coder-7B-Instruct produced the lowest average detection metrics among all evaluated models. These findings indicate that advanced LLMs are potentially promising to enhance the detection of SQL injection and complementary web application security.