LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL

• URL: http://arxiv.org/abs/2510.02350v1
• Date: Sat, 27 Sep 2025 15:08:43 GMT
• Title: LLMSQL: Upgrading WikiSQL for the LLM Era of Text-to-SQL
• Authors: Dzmitry Pihulski, Karol Charchut, Viktoria Novogrodskaia, Jan Kocoń,
• Abstract summary: We present LLM, a systematic revision and transformation of Wiki.<n>We classify these errors and implement automated methods for cleaning and re-annotation.<n>Rather than serving as an update, LLM is introduced as an LLM-ready benchmark.
• Score: 0.2799896314754614
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Converting natural language questions into SQL queries (Text-to-SQL) enables non-expert users to interact with relational databases and has long been a central task for natural language interfaces to data. While the WikiSQL dataset played a key role in early NL2SQL research, its usage has declined due to structural and annotation issues, including case sensitivity inconsistencies, data type mismatches, syntax errors, and unanswered questions. We present LLMSQL, a systematic revision and transformation of WikiSQL designed for the LLM era. We classify these errors and implement automated methods for cleaning and re-annotation. To assess the impact of these improvements, we evaluated multiple large language models (LLMs), including Gemma 3, LLaMA 3.2, Mistral 7B, gpt-oss 20B, Phi-3.5 Mini, Qwen 2.5, OpenAI o4-mini, DeepSeek R1 and others. Rather than serving as an update, LLMSQL is introduced as an LLM-ready benchmark: unlike the original WikiSQL, tailored for pointer-network models selecting tokens from input, LLMSQL provides clean natural language questions and full SQL queries as plain text, enabling straightforward generation and evaluation for modern natural language-to-SQL models.

Related papers

• DeKeyNLU: Enhancing Natural Language to SQL Generation through Task Decomposition and Keyword Extraction [46.422626657078666]
  We present DeKeyNLU, a novel dataset which contains 1,500 meticulously annotated QA pairs.<n>We propose DeKey, a RAG-based NL2 pipeline that employs three separate modules for user question understanding, entity retrieval, and generation.
  arXiv  Detail & Related papers  (2025-09-18T00:47:56Z)
• End-to-End Text-to-SQL with Dataset Selection: Leveraging LLMs for Adaptive Query Generation [6.5390580456423555]
  Traditional approaches model text-to- query as a direct translation task.<n>Recent advances in large language models (LLMs) have significantly improved translation accuracy.<n>We propose a three-stage end-to-end text-to-end framework to identify the user's intended database.
  arXiv  Detail & Related papers  (2025-08-08T15:16:36Z)
• A Survey of Text-to-SQL in the Era of LLMs: Where are we, and where are we going? [32.84561352339466]
  We provide a review of Text-to- translation techniques powered by Large Language Models (LLMs)<n>We discuss the research challenges and open problems of Text-to- evaluation in the LLMs era.
  arXiv  Detail & Related papers  (2024-08-09T14:59:36Z)
• Knowledge-to-SQL: Enhancing SQL Generation with Data Expert LLM [15.888784472807775]
  Existing methods rely on the comprehensive capability of large language models (LLMs) to generate queries. We propose the Knowledge-to- Data Expert framework, which employs tailored knowledge for all text-to- models.
  arXiv  Detail & Related papers  (2024-02-18T09:10:04Z)
• Interleaving Pre-Trained Language Models and Large Language Models for Zero-Shot NL2SQL Generation [23.519727682763644]
  ZeroNL2 is crucial in achieving natural language tosql that is adaptive to new environments. Existing approaches either fine-tune pretrained language models (PLMs) based on data or use prompts to guide fixed large language models (LLMs) such as ChatGPT. We propose a ZeroNL2 framework that combines the complementary advantages of PLMs and LLMs for supporting zero-shot NL2.
  arXiv  Detail & Related papers  (2023-06-15T06:50:51Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval [17.747079214502673]
  Text-to- is a task that converts a natural language question into a structured query language () to retrieve information from a database. In this paper, we propose an LLM-based framework for Text-to- which retrieves helpful demonstration examples to prompt LLMs. We design a de-semanticization mechanism that extracts question skeletons, allowing us to retrieve similar examples based on their structural similarity.
  arXiv  Detail & Related papers  (2023-04-26T06:02:01Z)
• XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing [70.40401197026925]
  In-context learning using large language models has recently shown surprising results for semantic parsing tasks. This work introduces the XRICL framework, which learns to retrieve relevant English exemplars for a given query. We also include global translation exemplars for a target language to facilitate the translation process for large language models.
  arXiv  Detail & Related papers  (2022-10-25T01:33:49Z)
• A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions [102.8606542189429]
  The goal of text-to-corpora parsing is to convert a natural language (NL) question to its corresponding structured query language () based on the evidences provided by databases. Deep neural networks have significantly advanced this task by neural generation models, which automatically learn a mapping function from an input NL question to an output query.
  arXiv  Detail & Related papers  (2022-08-29T14:24:13Z)
• Weakly Supervised Text-to-SQL Parsing through Question Decomposition [53.22128541030441]
  We take advantage of the recently proposed question meaning representation called QDMR. Given questions, their QDMR structures (annotated by non-experts or automatically predicted) and the answers, we are able to automatically synthesizesql queries. Our results show that the weakly supervised models perform competitively with those trained on NL- benchmark data.
  arXiv  Detail & Related papers  (2021-12-12T20:02:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.