Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data

• URL: http://arxiv.org/abs/2106.05006v1
• Date: Wed, 9 Jun 2021 12:09:51 GMT
• Title: Text-to-SQL in the Wild: A Naturally-Occurring Dataset Based on Stack Exchange Data
• Authors: Moshe Hazoom, Vibhor Malik and Ben Bogin
• Abstract summary: SEDE is a dataset with 12,023 pairs of utterances andsql queries collected from real usage on the Stack Exchange website. We show that these pairs contain a variety of real-world challenges which were rarely reflected so far in any other semantic parsing dataset.
• Score: 3.06261471569622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Most available semantic parsing datasets, comprising of pairs of natural utterances and logical forms, were collected solely for the purpose of training and evaluation of natural language understanding systems. As a result, they do not contain any of the richness and variety of natural-occurring utterances, where humans ask about data they need or are curious about. In this work, we release SEDE, a dataset with 12,023 pairs of utterances and SQL queries collected from real usage on the Stack Exchange website. We show that these pairs contain a variety of real-world challenges which were rarely reflected so far in any other semantic parsing dataset, propose an evaluation metric based on comparison of partial query clauses that is more suitable for real-world queries, and conduct experiments with strong baselines, showing a large gap between the performance on SEDE compared to other common datasets.

Related papers

• Semantic Parsing for Conversational Question Answering over Knowledge Graphs [63.939700311269156]
  We develop a dataset where user questions are annotated with Sparql parses and system answers correspond to execution results thereof. We present two different semantic parsing approaches and highlight the challenges of the task. Our dataset and models are released at https://github.com/Edinburgh/SPICE.
  arXiv  Detail & Related papers  (2023-01-28T14:45:11Z)
• Retrieval-based Disentangled Representation Learning with Natural Language Supervision [61.75109410513864]
  We present Vocabulary Disentangled Retrieval (VDR), a retrieval-based framework that harnesses natural language as proxies of the underlying data variation to drive disentangled representation learning. Our approach employ a bi-encoder model to represent both data and natural language in a vocabulary space, enabling the model to distinguish intrinsic dimensions that capture characteristics within data through its natural language counterpart, thus disentanglement.
  arXiv  Detail & Related papers  (2022-12-15T10:20:42Z)
• STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsing [64.80483736666123]
  We propose a novel pre-training framework STAR for context-dependent text-to- parsing. In addition, we construct a large-scale context-dependent text-to-the-art conversation corpus to pre-train STAR. Extensive experiments show that STAR achieves new state-of-the-art performance on two downstream benchmarks.
  arXiv  Detail & Related papers  (2022-10-21T11:30:07Z)
• xDBTagger: Explainable Natural Language Interface to Databases Using Keyword Mappings and Schema Graph [0.17188280334580192]
  Translating natural language queries into structured query language (NLQ) in interfaces to relational databases is a challenging task. We propose xDBTagger, an explainable hybrid translation pipeline that explains the decisions made along the way to the user both textually and visually. xDBTagger is effective in terms of accuracy and translates the queries more efficiently compared to other state-of-the-art pipeline-based systems up to 10000 times.
  arXiv  Detail & Related papers  (2022-10-07T18:17:09Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• KaggleDBQA: Realistic Evaluation of Text-to-SQL Parsers [26.15889661083109]
  We present KDBaggleQA, a new cross-domain evaluation dataset of real Web databases. We show that KDBaggleQA presents a challenge to state-of-the-art zero-shots but that a more realistic evaluation setting and creative use of associated database documentation boosts their accuracy by over 13.2%.
  arXiv  Detail & Related papers  (2021-06-22T00:08:03Z)
• Does Putting a Linguist in the Loop Improve NLU Data Collection? [34.34874979524489]
  Crowdsourcing NLP datasets contain systematic gaps and biases that are identified only after data collection is complete. We take natural language inference as a test case and ask whether it is beneficial to put a linguist in the loop' during data collection.
  arXiv  Detail & Related papers  (2021-04-15T00:31:10Z)
• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL [49.85635994436742]
  We include human in the loop and present a novel-independent interactive approach (PIIA) that interacts with users using multi-choice questions. PIIA is capable of enhancing the text-to-domain performance with limited interaction turns by using both simulation and human evaluation.
  arXiv  Detail & Related papers  (2020-11-09T02:14:33Z)
• Deep Graph Matching and Searching for Semantic Code Retrieval [76.51445515611469]
  We propose an end-to-end deep graph matching and searching model based on graph neural networks. We first represent both natural language query texts and programming language code snippets with the unified graph-structured data. In particular, DGMS not only captures more structural information for individual query texts or code snippets but also learns the fine-grained similarity between them.
  arXiv  Detail & Related papers  (2020-10-24T14:16:50Z)
• Comparative analysis of word embeddings in assessing semantic similarity of complex sentences [8.873705500708196]
  We study the sentences in existing benchmark datasets and analyze the sensitivity of various word embeddings with respect to the complexity of the sentences. The results show the increase in complexity of the sentences has a significant impact on the performance of the embedding models.
  arXiv  Detail & Related papers  (2020-10-23T19:55:11Z)
• ColloQL: Robust Cross-Domain Text-to-SQL Over Search Queries [10.273545005890496]
  We introduce data augmentation techniques and a sampling-based content-aware BERT model (ColloQL) ColloQL achieves 84.9% (execution) and 90.7% (execution) accuracy on the Wikilogical dataset.
  arXiv  Detail & Related papers  (2020-10-19T23:53:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.