ParSEval: Interactive Counterexample-Driven Evaluation for Text-to-SQL
TL;DR
This work presents ParSEval, an interactive demonstration system for configurable, counterexample-driven Text-to-SQL evaluation, which helps developers evaluate iterative model improvements on private benchmarks and bootstrap customized Text-to-SQL test suites without manually crafting diagnostic database instances.
Abstract
Evaluating Text-to-SQL systems on a single static database instance often misses semantic errors that arise only under specific data conditions, such as duplicates, NULLs, or integrity assumptions. As a result, developers may obtain misleading execution-accuracy results and struggle to understand their systems' true behavior. We present ParSEval, an interactive demonstration system for configurable, counterexample-driven Text-to-SQL evaluation. Given schemas, ground-truth queries, and model predictions, ParSEval automatically generates test databases under configurable integrity assumptions, evaluates query pairs across multiple instances, and exposes concrete witness instances when execution results differ. Through a live dashboard, users can compare model runs, inspect disagreement patterns under different evaluation settings, such as set versus bag semantics and key-foreign-key constraints, and drill down into individual failures through generated counterexamples. We demonstrate how ParSEval helps developers evaluate iterative model improvements on private benchmarks and bootstrap customized Text-to-SQL test suites without manually crafting diagnostic database instances.