Fundamental Challenges in Evaluating Text2SQL Solutions and Detecting Their Limitations

• URL: http://arxiv.org/abs/2501.18197v1
• Date: Thu, 30 Jan 2025 08:31:09 GMT
• Title: Fundamental Challenges in Evaluating Text2SQL Solutions and Detecting Their Limitations
• Abstract summary: We propose a unified taxonomy of all Text2 limitations that can lead to both prediction and evaluation errors.<n>We describe the causes of limitations with real-world examples and propose potential mitigation solutions for each category in the taxonomy.
• Score: 9.363953429870007
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this work, we dive into the fundamental challenges of evaluating Text2SQL solutions and highlight potential failure causes and the potential risks of relying on aggregate metrics in existing benchmarks. We identify two largely unaddressed limitations in current open benchmarks: (1) data quality issues in the evaluation data, mainly attributed to the lack of capturing the probabilistic nature of translating a natural language description into a structured query (e.g., NL ambiguity), and (2) the bias introduced by using different match functions as approximations for SQL equivalence. To put both limitations into context, we propose a unified taxonomy of all Text2SQL limitations that can lead to both prediction and evaluation errors. We then motivate the taxonomy by providing a survey of Text2SQL limitations using state-of-the-art Text2SQL solutions and benchmarks. We describe the causes of limitations with real-world examples and propose potential mitigation solutions for each category in the taxonomy. We conclude by highlighting the open challenges encountered when deploying such mitigation strategies or attempting to automatically apply the taxonomy.

Related papers

• Knowledge Distillation for Low-Resource Open-source Text-to-SQL Model [4.098313540411055]
  We propose a knowledge-aware Text-to-tt framework that constructs task-specific knowledge base and injects it into both training and inference.<n>This framework generates contextually grounded synthetic training data and enhances inference through targeted knowledge retrieval.
  arXiv  Detail & Related papers  (2026-05-13T07:54:13Z)
• CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems [13.595364881493943]
  Existing benchmarks typically assume a single source of ambiguity and rely on user interaction for resolution.<n>We introduce Clarity, a framework for automatically generating an NL2 benchmark with multi-faceted ambiguities and diverse user behaviors.<n>Our results highlight the need for more robust ambiguity detection and resolution in industry-grade NL2 systems.
  arXiv  Detail & Related papers  (2026-04-24T07:47:16Z)
• LatentRefusal: Latent-Signal Refusal for Unanswerable Text-to-SQL Queries [6.5781226398371615]
  Unanswerable and under user queries pose a major barrier to safe deployment in text-to-specified systems.<n>LatentRefusal is a latent-signal refusal mechanism that predicts answerability from hidden activations of a large language model.<n>We show that LatentRefusal improves average F1 to 88.5 percent on both backbones while adding approximately 2 milliseconds of probe overhead.
  arXiv  Detail & Related papers  (2026-01-15T13:48:22Z)
• Memo-SQL: Structured Decomposition and Experience-Driven Self-Correction for Training-Free NL2SQL [23.966546153810764]
  Existing NL2 systems rely on in-context learning with only correct examples.<n>We present Memo-correction, setting a new state of the art among open, zero-fine-tuning methods.
  arXiv  Detail & Related papers  (2026-01-15T02:42:05Z)
• FloodSQL-Bench: A Retrieval-Augmented Benchmark for Geospatially-Grounded Text-to-SQL [4.973502845481286]
  FLOOD-BENCH is a benchmark for the flood management domain that integrates heterogeneous datasets through key-based, spatial, and hybrid joins.<n>The benchmark captures realistic flood-related information needs by combining social, infrastructural, and hazard data layers.
  arXiv  Detail & Related papers  (2025-12-12T23:25:00Z)
• SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering [18.161591137171623]
  We introduce SCARE, a benchmark for evaluating methods that function as a post-hoc safety layer in EHR QA systems.<n>SCARE evaluates the joint task of (1) classifying question answerability (i.e., determining whether a question is answerable, ambiguous, or unanswerable) and (2) verifying or correcting candidatesql queries.
  arXiv  Detail & Related papers  (2025-11-13T06:35:29Z)
• Towards Context-aware Reasoning-enhanced Generative Searching in E-commerce [61.03081096959132]
  We propose a context-aware reasoning-enhanced generative search framework for better textbfunderstanding the complicated context.<n>Our approach achieves superior performance compared with strong baselines, validating its effectiveness for search-based recommendation.
  arXiv  Detail & Related papers  (2025-10-19T16:46:11Z)
• Who Gets Cited Most? Benchmarking Long-Context Language Models on Scientific Articles [81.89404347890662]
  SciTrek is a novel question-answering benchmark designed to evaluate the long-context reasoning capabilities of large language models (LLMs) using scientific articles.<n>Our analysis reveals systematic shortcomings in models' abilities to perform basic numerical operations and accurately locate specific information in long contexts.
  arXiv  Detail & Related papers  (2025-09-25T11:36:09Z)
• Text to Query Plans for Question Answering on Large Tables [4.917892629916144]
  We propose a novel framework that transforms natural language queries into query plans.<n>We enable complex analytical functions, such as principal component analysis and anomaly detection.<n>We validate our framework through experiments on both standard databases and large scientific tables.
  arXiv  Detail & Related papers  (2025-08-26T07:35:26Z)
• The Devil is in Fine-tuning and Long-tailed Problems:A New Benchmark for Scene Text Detection [2.4829769683482437]
  Scene text detection methods often fail to replicate such success in real-world scenarios.<n>We uncover two key factors contributing to this discrepancy through extensive experiments.<n>We propose a Long-Tailed Benchmark (LTB) to evaluate ability to handle a diverse range of long-tailed challenges.
  arXiv  Detail & Related papers  (2025-05-21T15:26:46Z)
• Domain Specific Question to SQL Conversion with Embedded Data Balancing Technique [0.0]
  This paper proposes two intermediations to improve the accuracy of structured query language models. The proposed solution achieved 10.98 percentage improvement in accuracy of the model performance compared to the state of the art model tested on Wiki dataset.
  arXiv  Detail & Related papers  (2025-03-28T08:58:14Z)
• SQLCritic: Correcting Text-to-SQL Generation via Clause-wise Critic [0.8098097078441623]
  We propose a novel approach combining structured execution feedback with a trained critic agent that provides detailed, interpretable critiques. This method effectively identifies and corrects both syntactic and semantic errors, enhancing accuracy and interpretability.
  arXiv  Detail & Related papers  (2025-03-11T02:52:39Z)
• Balancing Content Size in RAG-Text2SQL System [0.0]
  This research investigates the nuanced trade-off between document size and quality of retrieved documents. We explore the phenomenon of hallucinations in Text2 models, emphasizing the critical role of curated document presentation in minimizing errors. Our findings offer a roadmap for enhancing the robustness of RAG + Text2 systems, offering practical insights for real-world applications.
  arXiv  Detail & Related papers  (2025-01-28T06:06:28Z)
• FLEX: Expert-level False-Less EXecution Metric for Reliable Text-to-SQL Benchmark [8.445403382578167]
  This paper introduces FLEX (False-Lesscution EXecution), a novel approach to evaluating text-to-technical systems. Our metric improves agreement with human experts with comprehensive context and sophisticated criteria. This work contributes to a more accurate and nuanced evaluation of text-to-technical systems, potentially reshaping our understanding of state-of-the-art performance in this field.
  arXiv  Detail & Related papers  (2024-09-24T01:40:50Z)
• AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries [56.82807063333088]
  We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-open programs. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness) In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch.
  arXiv  Detail & Related papers  (2024-06-27T10:43:04Z)
• TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring [11.78795632771211]
  We introduce a novel benchmark designed to evaluate text-to- reliability as a model's ability to correctly handle any type of input question. We evaluate existing methods using a novel penalty-based scoring metric with two modeling approaches.
  arXiv  Detail & Related papers  (2024-03-23T16:12:52Z)
• How Well Do Text Embedding Models Understand Syntax? [50.440590035493074]
  The ability of text embedding models to generalize across a wide range of syntactic contexts remains under-explored. Our findings reveal that existing text embedding models have not sufficiently addressed these syntactic understanding challenges. We propose strategies to augment the generalization ability of text embedding models in diverse syntactic scenarios.
  arXiv  Detail & Related papers  (2023-11-14T08:51:00Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• Wav2SQL: Direct Generalizable Speech-To-SQL Parsing [55.10009651476589]
  Speech-to-Spider (S2Spider) aims to convert spoken questions intosql queries given databases. We propose the first direct speech-to-speaker parsing model Wav2 which avoids error compounding across cascaded systems. Experimental results demonstrate that Wav2 avoids error compounding and achieves state-of-the-art results by up to 2.5% accuracy improvement over the baseline.
  arXiv  Detail & Related papers  (2023-05-21T19:26:46Z)
• Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval [17.747079214502673]
  Text-to- is a task that converts a natural language question into a structured query language () to retrieve information from a database. In this paper, we propose an LLM-based framework for Text-to- which retrieves helpful demonstration examples to prompt LLMs. We design a de-semanticization mechanism that extracts question skeletons, allowing us to retrieve similar examples based on their structural similarity.
  arXiv  Detail & Related papers  (2023-04-26T06:02:01Z)
• Know What I don't Know: Handling Ambiguous and Unanswerable Questions for Text-to-SQL [36.5089235153207]
  Existing text-to-yourselfs generate a "plausible" query for an arbitrary user question. We propose a simple yet effective generation approach that automatically produces ambiguous and unanswerable examples. Experimental results show that our model achieves the best result on both real-world examples and generated examples.
  arXiv  Detail & Related papers  (2022-12-17T15:32:00Z)
• Towards Generalizable and Robust Text-to-SQL Parsing [77.18724939989647]
  We propose a novel TKK framework consisting of Task decomposition, Knowledge acquisition, and Knowledge composition to learn text-to- parsing in stages. We show that our framework is effective in all scenarios and state-of-the-art performance on the Spider, SParC, and Co. datasets.
  arXiv  Detail & Related papers  (2022-10-23T09:21:27Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers [26.15889661083109]
  We present KDBaggleQA, a new cross-domain evaluation dataset of real Web databases. We show that KDBaggleQA presents a challenge to state-of-the-art zero-shots but that a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%.
  arXiv  Detail & Related papers  (2021-06-22T00:08:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.