Text-to-SQL with Large Language Models: Challenges Revisited and New Dimensions
TL;DR
This position paper argues that each original challenge for LLM-based Text-to-SQL has given rise to a new dimension, and synthesizes these concerns and outlines a research agenda along three horizons, arguing for trustworthiness, interactivity, and economic sustainability as first-class concerns.
Abstract
: Building on a prior analysis that identified five key challenges for LLM-based Text-to-SQL (response time, scalability, hallucinations, dataset representativeness, and knowledge acquisition), this position paper argues that each original challenge has given rise to a new dimension: response-time and scalability pressures produced agentic approaches; the pursuit of accuracy on complex queries led to reasoning models; hallucination research exposed the deeper trustworthiness problem, including security; static-benchmark limitations motivated conversational evaluation; and scalability became an economic question, surfacing the open versus proprietary tension. The field has transformed: agentic architectures now surpass 81% execution accuracy on the BIRD test set, yet state-of-the-art systems remain no higher than 17% end-to-end success on multi-turn interactive benchmarks, and backdoor attacks succeed with less than 1% poisoned training data. We synthesize these concerns and outline a research agenda along three horizons, arguing for trustworthiness, interactivity, and economic sustainability as first-class concerns.