Analyzing the Effectiveness of Large Language Models on Text-to-SQL
Synthesis
- URL: http://arxiv.org/abs/2401.12379v1
- Date: Mon, 22 Jan 2024 22:05:42 GMT
- Title: Analyzing the Effectiveness of Large Language Models on Text-to-SQL
Synthesis
- Authors: Richard Roberson, Gowtham Kaki, Ashutosh Trivedi
- Abstract summary: This study investigates various approaches to using Large Language Models for Text-to- program synthesis.
The goal was to input a natural language question along with the database schema and output the correct SELECT query.
- Score: 4.412170175171256
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: This study investigates various approaches to using Large Language Models
(LLMs) for Text-to-SQL program synthesis, focusing on the outcomes and insights
derived. Employing the popular Text-to-SQL dataset, spider, the goal was to
input a natural language question along with the database schema and output the
correct SQL SELECT query. The initial approach was to fine-tune a local and
open-source model to generate the SELECT query. After QLoRa fine-tuning
WizardLM's WizardCoder-15B model on the spider dataset, the execution accuracy
for generated queries rose to a high of 61%. With the second approach, using
the fine-tuned gpt-3.5-turbo-16k (Few-shot) + gpt-4-turbo (Zero-shot error
correction), the execution accuracy reached a high of 82.1%. Of all the
incorrect queries, most can be categorized into a seven different categories of
what went wrong: selecting the wrong columns or wrong order of columns,
grouping by the wrong column, predicting the wrong values in conditionals,
using different aggregates than the ground truth, extra or too few JOIN
clauses, inconsistencies in the Spider dataset, and lastly completely incorrect
query structure. Most if not all of the queries fall into these categories and
it is insightful to understanding where the faults still lie with LLM program
synthesis and where they can be improved.
Related papers
- Fine-Tuning Language Models for Context-Specific SQL Query Generation [0.0]
This paper presents a novel approach to fine-tuning open-source large language models (LLMs) for the task of transforming natural language intosql queries.
We introduce models specialized in generatingsql queries, trained on synthetic datasets tailored to the Snowflake SQL and Google dialects.
Our methodology involves generating a context-specific dataset using GPT-4, then fine-tuning three open-source LLMs(Starcoder Plus, Code-Llama, and Mistral) employing the LoRa technique to optimize for resource constraints.
The fine-tuned models demonstrate superior performance in zero-shot settings compared to the baseline GP
arXiv Detail & Related papers (2023-12-04T18:04:27Z) - Benchmarking and Improving Text-to-SQL Generation under Ambiguity [25.283118418288293]
We develop a novel benchmark called AmbiQT where each text is interpretable as two plausible SQLs due to lexical and/or structural ambiguity.
We propose LogicalBeam, a new decoding algorithm that navigates thesql logic space using a blend of plan-based template generation and constrained infilling.
arXiv Detail & Related papers (2023-10-20T17:00:53Z) - SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs)
With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses.
With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
arXiv Detail & Related papers (2023-05-26T21:39:05Z) - UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
We introduce a UNIfied benchmark for Text-to-domain systems.
It is composed of publicly available text-to-domain datasets and 29K databases.
Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
arXiv Detail & Related papers (2023-05-25T17:19:52Z) - Wav2SQL: Direct Generalizable Speech-To-SQL Parsing [55.10009651476589]
Speech-to-Spider (S2Spider) aims to convert spoken questions intosql queries given databases.
We propose the first direct speech-to-speaker parsing model Wav2 which avoids error compounding across cascaded systems.
Experimental results demonstrate that Wav2 avoids error compounding and achieves state-of-the-art results by up to 2.5% accuracy improvement over the baseline.
arXiv Detail & Related papers (2023-05-21T19:26:46Z) - Know What I don't Know: Handling Ambiguous and Unanswerable Questions
for Text-to-SQL [36.5089235153207]
Existing text-to-yourselfs generate a "plausible" query for an arbitrary user question.
We propose a simple yet effective generation approach that automatically produces ambiguous and unanswerable examples.
Experimental results show that our model achieves the best result on both real-world examples and generated examples.
arXiv Detail & Related papers (2022-12-17T15:32:00Z) - Improving Text-to-SQL Semantic Parsing with Fine-grained Query
Understanding [84.04706075621013]
We present a general-purpose, modular neural semantic parsing framework based on token-level fine-grained query understanding.
Our framework consists of three modules: named entity recognizer (NER), neural entity linker (NEL) and neural entity linker (NSP)
arXiv Detail & Related papers (2022-09-28T21:00:30Z) - S$^2$SQL: Injecting Syntax to Question-Schema Interaction Graph Encoder
for Text-to-SQL Parsers [66.78665327694625]
We propose S$2$, injecting Syntax to question- encoder graph for Text-to- relational parsing.
We also employ the decoupling constraint to induce diverse edge embedding, which further improves the network's performance.
Experiments on the Spider and robustness setting Spider-Syn demonstrate that the proposed approach outperforms all existing methods when pre-training models are used.
arXiv Detail & Related papers (2022-03-14T09:49:15Z) - Dual Reader-Parser on Hybrid Textual and Tabular Evidence for Open
Domain Question Answering [78.9863753810787]
A large amount of world's knowledge is stored in structured databases.
query languages can answer questions that require complex reasoning, as well as offering full explainability.
arXiv Detail & Related papers (2021-08-05T22:04:13Z) - Data Agnostic RoBERTa-based Natural Language to SQL Query Generation [0.0]
The NL2 task aims at finding deep learning approaches to solve the problem converting by natural language questions into valid queries.
We have presented an approach with data privacy at its core.
Although we have not achieved state of the art results, we have eliminated the need for the table right from the training of the model.
arXiv Detail & Related papers (2020-10-11T13:18:46Z) - Bertrand-DR: Improving Text-to-SQL using a Discriminative Re-ranker [1.049360126069332]
We propose a novel discnative re-ranker to improve the performance of generative text-to-rimi models.
We analyze relative strengths of the text-to-rimi and re-ranker models for optimal performance.
We demonstrate the effectiveness of the re-ranker by applying it to two state-of-the-art text-to-rimi models.
arXiv Detail & Related papers (2020-02-03T04:52:47Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.