An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data

• URL: http://arxiv.org/abs/2603.07841v1
• Date: Sun, 08 Mar 2026 23:14:18 GMT
• Title: An Efficient and Effective Evaluator for Text2SQL Models on Unseen and Unlabeled Data
• Abstract summary: A persistent deployment challenge is to assess a newly trained Text2 system on an unseen and unlabeled dataset when no verified answers are available.<n>This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written labels are costly and time-consuming.<n>Fusion estimates accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets.
• Score: 41.195911171639445
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in large language models has strengthened Text2SQL systems that translate natural language questions into database queries. A persistent deployment challenge is to assess a newly trained Text2SQL system on an unseen and unlabeled dataset when no verified answers are available. This situation arises frequently because database content and structure evolve, privacy policies slow manual review, and carefully written SQL labels are costly and time-consuming. Without timely evaluation, organizations cannot approve releases or detect failures early. FusionSQL addresses this gap by working with any Text2SQL models and estimating accuracy without reference labels, allowing teams to measure quality on unseen and unlabeled datasets. It analyzes patterns in the system's own outputs to characterize how the target dataset differs from the material used during training. FusionSQL supports pre-release checks, continuous monitoring of new databases, and detection of quality decline. Experiments across diverse application settings and question types show that FusionSQL closely follows actual accuracy and reliably signals emerging issues. Our code is available at https://github.com/phkhanhtrinh23/FusionSQL.

Related papers

• ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL [51.98164069124653]
  ProSPy is a Profiling-driven framework for enterprise-scale Text-to-gnostic analysis.<n>ProSPy structures the reasoning process into four stages.<n>It first extracts fine-grained data evidence through automatic profiling.<n>It prunes large schemas into task-relevant contexts.
  arXiv  Detail & Related papers  (2026-06-04T08:13:05Z)
• GBV-SQL: Guided Generation and SQL2Text Back-Translation Validation for Multi-Agent Text2SQL [12.455525963127497]
  GBV- is a novel multi-agent framework that introduces Guided Generation with SQL2Text Back-translation Validation.<n>This mechanism uses a specialized agent to translate the generatedsql back into natural language, which verifies its logical alignment with the original question.<n>We introduce a formal typology for "Gold Errors", which are pervasive flaws in the ground-truth and demonstrate how they obscure true model performance.
  arXiv  Detail & Related papers  (2025-09-16T03:21:12Z)
• AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL [0.9217021281095907]
  We demonstrate Ambi, an interactive system that automatically detects query ambiguities and guides users through multiple-choice questions to clarify their intent.<n>Ambi achieves 87.2% in ambiguity detection and improves exact accuracy by 50% when integrated with Text-to- dataset systems.
  arXiv  Detail & Related papers  (2025-08-21T06:10:28Z)
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows [64.94146689665628]
  Spider 2.0 is an evaluation framework for real-world text-to-sql problems derived from enterprise-level database use cases.<n>The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake.<n>We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-levels.
  arXiv  Detail & Related papers  (2024-11-12T12:52:17Z)
• TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring [11.78795632771211]
  We introduce a novel benchmark designed to evaluate text-to- reliability as a model's ability to correctly handle any type of input question. We evaluate existing methods using a novel penalty-based scoring metric with two modeling approaches.
  arXiv  Detail & Related papers  (2024-03-23T16:12:52Z)
• Evaluating the Data Model Robustness of Text-to-SQL Systems Based on Real User Queries [5.8226163227899645]
  This paper is the first in-depth evaluation of the data model robustness of Text-to-- systems in practice.<n>It is based on a real-world deployment of FootballDB, a system that was deployed over a 9 month period in the context of the FIFA World Cup 2022.<n>All of our data is based on real user questions that were asked live to the system. We manually labeled and translated a subset of these questions for three different data models.
  arXiv  Detail & Related papers  (2024-02-13T10:28:57Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• Error Detection for Text-to-SQL Semantic Parsing [18.068244400731366]
  Modern text-to- semantics are often over-confident, casting doubt on their trustworthiness when deployed for real use. We propose a-independent error detection model for text-to- semantic parsing.
  arXiv  Detail & Related papers  (2023-05-23T04:44:22Z)
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs [89.68522473384522]
  We present Bird, a big benchmark for large-scale database grounded in text-to-efficient tasks. Our emphasis on database values highlights the new challenges of dirty database contents. Even the most effective text-to-efficient models, i.e. ChatGPT, achieves only 40.08% in execution accuracy.
  arXiv  Detail & Related papers  (2023-05-04T19:02:29Z)
• A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions [102.8606542189429]
  The goal of text-to-corpora parsing is to convert a natural language (NL) question to its corresponding structured query language () based on the evidences provided by databases. Deep neural networks have significantly advanced this task by neural generation models, which automatically learn a mapping function from an input NL question to an output query.
  arXiv  Detail & Related papers  (2022-08-29T14:24:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.