Synthesizing Text-to-SQL Data from Weak and Strong LLMs

• URL: http://arxiv.org/abs/2408.03256v1
• Date: Tue, 6 Aug 2024 15:40:32 GMT
• Title: Synthesizing Text-to-SQL Data from Weak and Strong LLMs
• Authors: Jiaxi Yang, Binyuan Hui, Min Yang, Jian Yang, Junyang Lin, Chang Zhou,
• Abstract summary: The capability gap between open-source and closed-source large language models (LLMs) remains a challenge in text-to- tasks. We introduce a synthetic data approach that combines data produced by larger, more powerful models with error information data generated by smaller, not well-aligned models.
• Score: 68.69270834311259
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The capability gap between open-source and closed-source large language models (LLMs) remains a challenge in text-to-SQL tasks. In this paper, we introduce a synthetic data approach that combines data produced by larger, more powerful models (strong models) with error information data generated by smaller, not well-aligned models (weak models). The method not only enhances the domain generalization of text-to-SQL models but also explores the potential of error data supervision through preference learning. Furthermore, we employ the synthetic data approach for instruction tuning on open-source LLMs, resulting SENSE, a specialized text-to-SQL model. The effectiveness of SENSE is demonstrated through state-of-the-art results on the SPIDER and BIRD benchmarks, bridging the performance gap between open-source models and methods prompted by closed-source models.

Related papers

• Can Models Help Us Create Better Models? Evaluating LLMs as Data Scientists [41.94295877935867]
  We present a benchmark for large language models designed to tackle one of the most knowledge-intensive tasks in data science. We demonstrate that the FeatEng of our proposal can cheaply and efficiently assess the broad capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-10-30T17:59:01Z)
• MSc-SQL: Multi-Sample Critiquing Small Language Models For Text-To-SQL Translation [10.205010004198757]
  Text-to-generation enables non-experts to interact with databases via natural language. Recent advances on large closed-source models like GPT-4 present challenges in accessibility, privacy, and latency. We focus on developing small, efficient, and open-source text-to-generation models.
  arXiv  Detail & Related papers  (2024-10-16T18:03:24Z)
• Unlocking the Potential of Model Merging for Low-Resource Languages [66.7716891808697]
  Adapting large language models to new languages typically involves continual pre-training (CT) followed by supervised fine-tuning (SFT) We propose model merging as an alternative for low-resource languages, combining models with distinct capabilities into a single model without additional training. Experiments based on Llama-2-7B demonstrate that model merging effectively endows LLMs for low-resource languages with task-solving abilities, outperforming CT-then-SFT in scenarios with extremely scarce data.
  arXiv  Detail & Related papers  (2024-07-04T15:14:17Z)
• CHESS: Contextual Harnessing for Efficient SQL Synthesis [1.9506402593665235]
  We propose a new pipeline that retrieves relevant data and context, selects an efficient schema, and synthesizes correct and efficient queries. Our method achieves new state-of-the-art performance on the cross-domain challenging BIRD dataset.
  arXiv  Detail & Related papers  (2024-05-27T01:54:16Z)
• CodeS: Towards Building Open-source Language Models for Text-to-SQL [42.11113113574589]
  We introduce CodeS, a series of pre-trained language models with parameters ranging from 1B to 15B. CodeS is a fully open language model, which achieves superior accuracy with much smaller parameter sizes. We conduct comprehensive evaluations on multiple datasets, including the widely used Spider benchmark.
  arXiv  Detail & Related papers  (2024-02-26T07:00:58Z)
• DTS-SQL: Decomposed Text-to-SQL with Small Large Language Models [7.388002745070808]
  We introduce a novel two-stage fine-tuning approach that decomposes the task into two simpler tasks. We show that this approach improves execution accuracy by 3 to 7 percent.
  arXiv  Detail & Related papers  (2024-02-02T03:21:00Z)
• Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation [76.76046657162306]
  Large language models (LLMs) have emerged as a new paradigm for Text-to- task. Large language models (LLMs) have emerged as a new paradigm for Text-to- task.
  arXiv  Detail & Related papers  (2023-08-29T14:59:54Z)
• Importance of Synthesizing High-quality Data for Text-to-SQL Parsing [71.02856634369174]
  State-of-the-art text-to-weighted algorithms did not further improve on popular benchmarks when trained with augmented synthetic data. We propose a novel framework that incorporates key relationships from schema, imposes strong typing, and schema-weighted column sampling.
  arXiv  Detail & Related papers  (2022-12-17T02:53:21Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• Learning to Synthesize Data for Semantic Parsing [57.190817162674875]
  We propose a generative model which models the composition of programs and maps a program to an utterance. Due to the simplicity of PCFG and pre-trained BART, our generative model can be efficiently learned from existing data at hand. We evaluate our method in both in-domain and out-of-domain settings of text-to-Query parsing on the standard benchmarks of GeoQuery and Spider.
  arXiv  Detail & Related papers  (2021-04-12T21:24:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.