Confidence Estimation for Error Detection in Text-to-SQL Systems

• URL: http://arxiv.org/abs/2501.09527v1
• Date: Thu, 16 Jan 2025 13:23:07 GMT
• Title: Confidence Estimation for Error Detection in Text-to-SQL Systems
• Authors: Oleg Somov, Elena Tutubalina,
• Abstract summary: This study investigates the integration of selective classifiers into Text-to-learning systems. We show that encoder-decoder T5 is better calibrated than in-context GPT 4 and decoder-only Llama 3. In terms of error detection, selective classifier with a higher probability detects errors associated with irrelevant questions rather than incorrect query generations.
• Score: 5.636160825241556
• License:
• Abstract: Text-to-SQL enables users to interact with databases through natural language, simplifying the retrieval and synthesis of information. Despite the success of large language models (LLMs) in converting natural language questions into SQL queries, their broader adoption is limited by two main challenges: achieving robust generalization across diverse queries and ensuring interpretative confidence in their predictions. To tackle these issues, our research investigates the integration of selective classifiers into Text-to-SQL systems. We analyse the trade-off between coverage and risk using entropy based confidence estimation with selective classifiers and assess its impact on the overall performance of Text-to-SQL models. Additionally, we explore the models' initial calibration and improve it with calibration techniques for better model alignment between confidence and accuracy. Our experimental results show that encoder-decoder T5 is better calibrated than in-context-learning GPT 4 and decoder-only Llama 3, thus the designated external entropy-based selective classifier has better performance. The study also reveal that, in terms of error detection, selective classifier with a higher probability detects errors associated with irrelevant questions rather than incorrect query generations.

Related papers

• Reliable Text-to-SQL with Adaptive Abstention [21.07332675929629]
  We present a novel framework that enhances query generation reliability by incorporating abstention and human-in-the-loop mechanisms. We validate our approach through comprehensive experiments on the BIRD benchmark, demonstrating significant improvements in robustness and reliability.
  arXiv  Detail & Related papers  (2025-01-18T19:36:37Z)
• Text-to-SQL Calibration: No Need to Ask -- Just Rescale Model Probabilities [20.606333546028516]
  We show that a straightforward baseline -- deriving confidence from the model's full-sequence probability -- outperforms recent methods. Our comprehensive evaluation, conducted across two widely-used Text-to-checking benchmarks and multiple architectures, provides valuable insights into the effectiveness of various calibration strategies.
  arXiv  Detail & Related papers  (2024-11-23T19:20:24Z)
• Epistemic Integrity in Large Language Models [11.173637560124828]
  Large language models are increasingly relied upon sources of information, but their propensity for false or misleading statements poses high risks for users and society. In this paper, we confront the critical problem of miscalibration where a model's linguistic assertiveness fails to reflect its true internal certainty. We introduce a new human misalignment evaluation and a novel method for measuring the linguistic assertiveness of Large Language Models.
  arXiv  Detail & Related papers  (2024-11-10T17:10:13Z)
• Context-Aware SQL Error Correction Using Few-Shot Learning -- A Novel Approach Based on NLQ, Error, and SQL Similarity [0.0]
  This paper introduces a novel few-shot learning-based approach for error correction insql generation. It enhances the accuracy of generated queries by selecting the most suitable few-shot error correction examples for a given natural language question (NLQ) In experiments with the open-source dataset, the proposed model offers a 39.2% increase in fixing errors with no error correction and a 10% increase from a simple error correction method.
  arXiv  Detail & Related papers  (2024-10-11T18:22:08Z)
• Bring Your Own Data! Self-Supervised Evaluation for Large Language Models [52.15056231665816]
  We propose a framework for self-supervised evaluation of Large Language Models (LLMs) We demonstrate self-supervised evaluation strategies for measuring closed-book knowledge, toxicity, and long-range context dependence. We find strong correlations between self-supervised and human-supervised evaluations.
  arXiv  Detail & Related papers  (2023-06-23T17:59:09Z)
• Transformer-based approaches to Sentiment Detection [55.41644538483948]
  We examined the performance of four different types of state-of-the-art transformer models for text classification. The RoBERTa transformer model performs best on the test dataset with a score of 82.6% and is highly recommended for quality predictions.
  arXiv  Detail & Related papers  (2023-03-13T17:12:03Z)
• Dr.Spider: A Diagnostic Evaluation Benchmark towards Text-to-SQL Robustness [115.66421993459663]
  Recent studies reveal that text-to- models are vulnerable to task-specific perturbations. We propose a comprehensive robustness benchmark based on Spider to diagnose the model. We conduct a diagnostic study of the state-of-the-art models on the set.
  arXiv  Detail & Related papers  (2023-01-21T03:57:18Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• Improving Classifier Training Efficiency for Automatic Cyberbullying Detection with Feature Density [58.64907136562178]
  We study the effectiveness of Feature Density (FD) using different linguistically-backed feature preprocessing methods. We hypothesise that estimating dataset complexity allows for the reduction of the number of required experiments. The difference in linguistic complexity of datasets allows us to additionally discuss the efficacy of linguistically-backed word preprocessing.
  arXiv  Detail & Related papers  (2021-11-02T15:48:28Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.