Decisions are being made based on data, and the demands for easy-to-use interfaces that enable users to simply type in a question to a relational database without having to be a Structured Query Language (SQL) expert have increased. Although the development of Large Language Models (LLMs) has seen remarkable advancements in the past few years, and has led to the emergence of LLMs that have significantly improved the generation of SQL from natural language (Text-to-SQL)[1], there is a relative lack of systematic comparisons between the latest commercial LLMs that has been tested against a series of graded SQL complexity levels with both syntactic and execution-based metrics. This research aims to assess and contrast ChatGPT, Google Gemini, DeepSeek, and Microsoft Copilot for their accuracy and efficacy in creating SQL queries from natural language queries. In a quantitative experimental design, 100 natural language questions were given using a purpose-built benchmark, with six types of queries that included simple retrieval, filtering, joins, aggregation, GROUP BY and HAVING, and nested subqueries. The results of the models were assessed using the manually written and validated ground truth query, with the use of normalised Exact Match Accuracy (EMA) and Execution Accuracy (EXA). The best overall results (EMA 97%, EXA 97%) were obtained by ChatGPT. The Copilot scored EMA 56% and EXA 96%, DeepSeek EMA 55% and EXA 93% and Gemini the lowest with a score of EMA 36% and EXA 87%. The large margin between EMA and EXA for the models (+51 points, +40 points, +38 points) demonstrates the significant shortcomings of exact-match scoring when it comes to assessing real-world skills for producing valid but structurally different SQL. The study empirically demonstrates comparative behaviours of the current LLMs, and suggests the use of execution-aware evaluation in the studies and deployment of Text-to-SQL.
Bharath Kumar N, T L Manasa· International journal of com...· 0 citations
Large Language Models (LLMs) have revolutionized software development, from analyzing code and generating suggestions to detecting bugs and errors, and even creating entire programs. Despite these advances, existing AI-driven code review solutions still provide a one-size-fits-all approach to code review with overall feedback and suggestions, often of a non-specific nature. This restriction promotes modular architectures which would be able to provide specific and direct code quality reports. This paper presents the AgentCodeReview system, a multi-agent system that is able to conduct explainable code review and automated bug repair by leveraging software engineering agents with different code review tasks. There would be five independent entities, each one to be able to review code, analyze security, evaluate performance, document it and be able to automatically fix bugs. They run parallelly under the guidance of a centralized orchestration layer that collects the results from the analytical agents, calculates software quality scores and creates comprehensive HTML and PDF reports. Moreover, a Streamlit-based web interface was created that allows the interactive visualization of the results of the analysis and interactive entry of the input values. A set of twenty python programs was created to test the framework for effectiveness, consisting of a variety of runtime errors, security flaws, performance issues, documentation issues and a mixture of these types of errors. Two metrics, namely execution time and qualitative assessment were used to compare the proposed multi-agent framework with a single-agent framework as baseline. Experimental results demonstrated the benchmark execution success rate was 95%, while the multi-agent architecture provided more structured, explainable and domain specific feedback than the single agent. The extra computational cost of the coordinated analyses was acceptable for software quality assessment tasks because of the resulting interpretability and modularity. Through implementation and experiments, the results demonstrate AgentCodeReview's utility and extensibility to the field of explainable AI in software quality assurance. The proposed architecture can be expanded to other programming languages, integrated into the industrial development flow, and enhanced with the advanced LLMs for scalable intelligent code review.
Bharath Kumar N, T L Manasa· International journal of com...· 0 citations