Automating End-to-End Hybrid Query Processing: Benchmark, Solution, and Insights

ExpeSQL: An Efficient, Experience-Guided Decompositional Search Framework for Text-to-SQL

GQLBench: A Large-Scale Cross-Domain, Cross-Dialect Benchmark for NL2GQL

. Abstract Despite growing interest in NL2GQL, benchmarking progress has been constrained by the lack of resources that are simultaneously large-scale, cross-domain, and cross-dialect. To address this gap, we present GQLBench , a new benchmark built through an automated and scalable framework that integrates NL2SQL-to-NL2GQL conversion with graph-native data generation. GQLBench supports execution-based evaluation on both Cypher and ISO GQL, covering hundreds of graph databases and over 20k natural language questions for each dialect. By combining converted data from mature NL2SQL resources with synthetic graph-specific queries, it captures both schema diversity from real-world relational sources and graph-native reasoning challenges, including long paths and cycles. Beyond overall performance comparison, GQLBench also enables fine-grained evaluation across dialects, graph patterns, and query complexity. Experiments on advanced LLMs show that even strong proprietary models struggle on GQLBench, with gemini-3-flash achieving only 35.40% average execution accuracy across the two di-alects. Our data and code are available at https://github.com/qxssadf/GQLBench .

Eagle: Leveraging Operations Documents for Comprehensive Benchmark Question Generation

Eagle, a comprehensive benchmarking framework tailored for evaluating OpsLLMs, delivers a deployable foundation for advancing large-model applications in AIOps and open-source the framework and dataset to foster community adoption and reproducibility.

TEFD: A Benchmark for Natural Language to Flux Query Generation in Time-Series Databases

The proliferation of IoT and real-time monitoring has established Time-Series Databases (TSDBs) like InfluxDB as critical infrastructure. However, their functional query languages (e.g., Flux) present a steep learning curve, hindering data accessibility for non-experts. While Natural Language Interfaces (NLIs) offer a potential solution, the domain of Text-to-Flux is stalled by a critical bottleneck: the complete absence of diverse, high-quality paired benchmarks. To address this, we introduce the Text-to-Flux task and propose FluxEngine, a novel automated pipeline for dataset construction. Unlike static generation methods used in Text-to-SQL, our framework features a Self-Sustaining Live Data Context that utilizes background tasks to perpetually generate fresh data, ensuring that queries involving relative time windows (e.g., ''past hour'') remain executable and valid indefinitely. Using this framework, we construct and release TEFD (Text-to-Flux Dataset), the first large-scale benchmark for this task. We further define execution-based evaluation metrics tailored for time-series validity. This work provides the essential infrastructure to foster future research in NLI for TSDBs. To facilitate reproducibility and future research, our dataset and benchmark code are publicly available at https://github.com/gta886/TEFD-Benchmark.

GradeSQL: Outcome reward models for intelligent Text-to-SQL generation from LLMs

As Large Language Models (LLMs) become foundational to next-generation Intelligent Information Systems, the bridge between natural language interfaces and structured database systems remains a critical bottleneck. While Text-to-SQL generation enables cooperative support for complex query formulation, ensuring the reliability of these generated queries at inference time is a central challenge. Conventional methods rely on coarse execution-based signals, which may limit their ability to capture the nuanced semantic alignment required for high-stakes database environments. In this work, we propose the use of Outcome Reward Models (ORMs) as a fine-grained, probabilistic feedback mechanism for test-time verification in Text-to-SQL tasks. We introduce GradeSQL, a framework for training task-specific ORMs that assign scalar utility scores to candidate SQL queries based on their semantic correctness and alignment with database schema. Our approach is evaluated on the BIRD and Spider benchmarks across multiple open-source LLM families. Experimental results demonstrate that ORM-based verification consistently outperforms traditional execution-based heuristics.

MARS: Multi-hop Adaptive Retrieval and SPARQL Generation for KGQA

Large language models (LLMs) have demonstrated strong reasoning performance, but their tendency to hallucinate limits their reliability in knowledge-intensive tasks requiring up-to-date and grounded information. Combining knowledge graphs (KGs) with LLMs facilitates the use of explicit symbolic knowledge that can be continuously updated without costly fine-tuning, while benefiting from rapidly advancing LLM reasoning. We propose MARS, a scalable knowledge graph question answering (KGQA) approach that requires no model fine-tuning. Rather than relying on open-ended agentic exploration, MARS performs a structured retrieval procedure that links question entities to the KG and iteratively retrieves relevant next-hop information. At each step, MARS decides whether to continue graph traversal or to generate the final SPARQL query, allowing the model to adapt the retrieval depth to the question while keeping the overall pipeline more predictable than fully agentic approaches. We evaluate MARS on three established KGQA benchmarks across several LLMs and settings, including multilingual evaluation, and provide insights through ablation studies and error analysis. Our approach achieves competitive performance relative to state-of-the-art methods while remaining efficient and scalable. The evaluation results, code and resources are publicly available: https://github.com/dice-group/mars-kgqa.