: Building on a prior analysis that identified five key challenges for LLM-based Text-to-SQL (response time, scalability, hallucinations, dataset representativeness, and knowledge acquisition), this position paper argues that each original challenge has given rise to a new dimension: response-time and scalability pressures produced agentic approaches; the pursuit of accuracy on complex queries led to reasoning models; hallucination research exposed the deeper trustworthiness problem, including security; static-benchmark limitations motivated conversational evaluation; and scalability became an economic question, surfacing the open versus proprietary tension. The field has transformed: agentic architectures now surpass 81% execution accuracy on the BIRD test set, yet state-of-the-art systems remain no higher than 17% end-to-end success on multi-turn interactive benchmarks, and backdoor attacks succeed with less than 1% poisoned training data. We synthesize these concerns and outline a research agenda along three horizons, arguing for trustworthiness, interactivity, and economic sustainability as first-class concerns.
Luca Sala, Giovanni Sullutrone, Sonia Bergamaschi· Proceedings of the 15th Inte...· 0 citations
Large Language Models (LLMs) demonstrate high performance on curated Text-to-SQL benchmarks; nevertheless, real-world users frequently pose ambiguous or unanswerable questions that current systems handle poorly. Three interconnected gaps hinder progress: incomplete taxonomies, realistic benchmark generation for real-world settings, and static user interaction. We address all of the above issues through three contributions: (1) a unified taxonomy of 8 categories covering ambiguous and unanswerable questions; (2) a multi-agent generation pipeline with a two-stage process (NLQ generation followed by SQL grounding) and an explicit Category Conformance validation stage, producing questions from arbitrary databases validated by a council of local open-source models; and (3) ABISS (Ambiguity Benchmark using Interaction-Simulated Sessions), a dynamic simulation environment where Text-to-SQL agents interact with style-aware simulated users across multi-turn dialogues. Experiments with eight open-source models on ABISS-BIRD and ABISS-Spider reveal two fundamental bottlenecks. The first is subcategory classification: models detect that a question is problematic yet struggle to pinpoint the specific subcategory. The second is clarification-conditioned SQL generation: even after receiving useful user information, models often still fail in the final resolution step. Providing the ground truth category yields large gains in both execution and feedback across both datasets, yet ambiguous-question execution remains low even under oracle category labels. We release our code for data generation and benchmark on GitHub (https://github.com/giosullutrone/ABISS-Evaluating-Text-to-SQL-Systems-Through-Agent-Interaction).
Giovanni Sullutrone, Luca Sala, Sania Aftar et al.· 0 citations