Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation

• URL: http://arxiv.org/abs/2405.15307v1
• Date: Fri, 24 May 2024 07:51:08 GMT
• Title: Before Generation, Align it! A Novel and Effective Strategy for Mitigating Hallucinations in Text-to-SQL Generation
• Authors: Ge Qu, Jinyang Li, Bowen Li, Bowen Qin, Nan Huo, Chenhao Ma, Reynold Cheng,
• Abstract summary: Large Language Models (LLMs) driven by In-Context Learning (ICL) have significantly improved the performance of text-to-. Previous methods generally employ a two-stage reasoning framework, namely 1) linking schema and 2) logical synthesis, making the framework not only effective but also interpretable. Despite these advancements, the inherent bad nature of the generalization of LLMs often results in hallucinations, which limits the full potential of LLMs. In this work, we first identify and categorize the common types of hallucinations at each stage in text-to-. We then introduce a novel strategy, Task Alignment (TA),
• Score: 21.58204328067628
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large Language Models (LLMs) driven by In-Context Learning (ICL) have significantly improved the performance of text-to-SQL. Previous methods generally employ a two-stage reasoning framework, namely 1) schema linking and 2) logical synthesis, making the framework not only effective but also interpretable. Despite these advancements, the inherent bad nature of the generalization of LLMs often results in hallucinations, which limits the full potential of LLMs. In this work, we first identify and categorize the common types of hallucinations at each stage in text-to-SQL. We then introduce a novel strategy, Task Alignment (TA), designed to mitigate hallucinations at each stage. TA encourages LLMs to take advantage of experiences from similar tasks rather than starting the tasks from scratch. This can help LLMs reduce the burden of generalization, thereby mitigating hallucinations effectively. We further propose TA-SQL, a text-to-SQL framework based on this strategy. The experimental results and comprehensive analysis demonstrate the effectiveness and robustness of our framework. Specifically, it enhances the performance of the GPT-4 baseline by 21.23% relatively on BIRD dev and it yields significant improvements across six models and four mainstream, complex text-to-SQL benchmarks.

Related papers

• Hallucination Detection for LLM-based Text-to-SQL Generation via Two-Stage Metamorphic Testing [8.942002314582789]
  Large language models (LLMs) generate hallucinations, i.e.,unrealistic or illogical content.<n>We propose a novel hallucination detection method based on metamorphic testing (MT) that does not require standard answers.<n>Trials demonstrate our method's superior performance in terms of the F1-score, which ranges from 69.36% to 82.76%.
  arXiv  Detail & Related papers  (2025-12-24T04:04:26Z)
• Think2SQL: Reinforce LLM Reasoning Capabilities for Text2SQL [16.02851357789021]
  This paper investigates the influence of reasoning on Text2 performance on four benchmark datasets. It considers the following ZSL settings: general-purpose reasoning or not; (2) SFT, with and without task-specific reasoning traces; (3) RL, exploring the use of different rewarding functions. The results show that general-purpose reasoning under ZSL proves to be ineffective in tackling complex Text2 cases.
  arXiv  Detail & Related papers  (2025-04-21T13:05:26Z)
• OpenSearch-SQL: Enhancing Text-to-SQL with Dynamic Few-shot and Consistency Alignment [6.2089733671434875]
  We propose OpenSearch-, which divides the Text-to-agent task into four main modules: Preprocessing, Extraction, Generation, and Refinement, along with an Alignment module based on consistency alignment mechanism. These methods have significantly improved the performance of LLMs in the Text-to-agent task. Experimental results show that OpenSearch- achieves an execution accuracy(EX) of 69.3% on the BIRD development set, 72.28% on the test set, and a reward-based efficiency score (R-VES) of 69.3, with all three metrics ranking first at the time of submission.
  arXiv  Detail & Related papers  (2025-02-19T07:51:50Z)
• MCTS-SQL: Light-Weight LLMs can Master the Text-to-SQL through Monte Carlo Tree Search [1.166711394125328]
  Text-to-OTA is a fundamental yet challenging task in the NLP area.<n>We propose MCTS-OTA, a novel framework that uses Monte Carlo Tree Search.<n>We propose a token-level prefixcache mechanism that stores prior information during iterations.
  arXiv  Detail & Related papers  (2025-01-28T00:52:23Z)
• ROUTE: Robust Multitask Tuning and Collaboration for Text-to-SQL [42.019659095480726]
  We propose a novel RObust mUltitask Tuning and collaboration mEthod (ROUTE) to improve the comprehensive capabilities of open-source LLMs for Text2. Our approach begins with multi-task supervised fine-tuning (SFT) using various synthetic training data related tosql generation. We also introduce a Multitask Collaboration Prompting (MCP) strategy to reduce hallucinations duringsql generation.
  arXiv  Detail & Related papers  (2024-12-13T13:41:18Z)
• Leveraging Prior Experience: An Expandable Auxiliary Knowledge Base for Text-to-SQL [0.5735035463793009]
  Large Language Models (LLMs) exhibit impressive problem-solving skills across many tasks, but they still underperform compared to humans in various downstream applications, such as text-to-context. We propose LPE-Leveraging, a novel framework designed to augment LLMs by enabling continual learning without requiring fine-tuning. Our experimental results demonstrate that this continual learning approach yields substantial performance gains.
  arXiv  Detail & Related papers  (2024-11-20T12:03:17Z)
• RSL-SQL: Robust Schema Linking in Text-to-SQL Generation [51.00761167842468]
  We propose a novel framework called RSL- that combines bidirectional schema linking, contextual information augmentation, binary selection strategy, and multi-turn self-correction. benchmarks demonstrate that our approach achieves SOTA execution accuracy among open-source solutions, with 67.2% on BIRD and 87.9% on GPT-4ocorrection. Our approach outperforms a series of GPT-4 based Text-to-Seek systems when adopting DeepSeek (much cheaper) with same intact prompts.
  arXiv  Detail & Related papers  (2024-10-31T16:22:26Z)
• PTD-SQL: Partitioning and Targeted Drilling with LLMs in Text-to-SQL [54.304872649870575]
  Large Language Models (LLMs) have emerged as powerful tools for Text-to-sense tasks. In this study, we propose that employing query group partitioning allows LLMs to focus on learning the thought processes specific to a single problem type.
  arXiv  Detail & Related papers  (2024-09-21T09:33:14Z)
• RB-SQL: A Retrieval-based LLM Framework for Text-to-SQL [48.516004807486745]
  Large language models (LLMs) with in-context learning have significantly improved the performance of text-to- task. We propose RB-, a novel retrieval-based framework for in-context prompt engineering. Experiment results demonstrate that our model achieves better performance than several competitive baselines on public datasets BIRD and Spider.
  arXiv  Detail & Related papers  (2024-07-11T08:19:58Z)
• PET-SQL: A Prompt-Enhanced Two-Round Refinement of Text-to-SQL with Cross-consistency [19.067737007347613]
  Methods achieve new SOTA results on the Spider benchmark, with an execution accuracy of 87.6%. Our methods achieve new SOTA results on the Spider benchmark, with an execution accuracy of 87.6%.
  arXiv  Detail & Related papers  (2024-03-13T02:32:41Z)
• Benchmarking the Text-to-SQL Capability of Large Language Models: A Comprehensive Evaluation [33.41556606816004]
  Large Language Models (LLMs) have emerged as a powerful tool in advancing the Text-to- task. There is still no consensus on the optimal prompt templates and design frameworks. Existing benchmarks inadequately explore the performance of LLMs across the various sub-tasks of the Text-to- process.
  arXiv  Detail & Related papers  (2024-03-05T13:23:48Z)
• Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation [76.76046657162306]
  Large language models (LLMs) have emerged as a new paradigm for Text-to- task. Large language models (LLMs) have emerged as a new paradigm for Text-to- task.
  arXiv  Detail & Related papers  (2023-08-29T14:59:54Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• Towards Generalizable and Robust Text-to-SQL Parsing [77.18724939989647]
  We propose a novel TKK framework consisting of Task decomposition, Knowledge acquisition, and Knowledge composition to learn text-to- parsing in stages. We show that our framework is effective in all scenarios and state-of-the-art performance on the Spider, SParC, and Co. datasets.
  arXiv  Detail & Related papers  (2022-10-23T09:21:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.