SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads

• URL: http://arxiv.org/abs/2507.06192v1
• Date: Tue, 08 Jul 2025 17:20:34 GMT
• Title: SQLBarber: A System Leveraging Large Language Models to Generate Customized and Realistic SQL Workloads
• Authors: Jiale Lao, Immanuel Trummer,
• Abstract summary: sqlBarber is a system based on Large Language Models (LLMs) to generate customized and realisticsql workloads.<n>It reduces query generation time by one to three orders of magnitude, and significantly improves alignment with the target cost distribution.<n>We construct and open-source ten benchmarks of varying difficulty levels and target query cost distributions based on real-world statistics from Snowflake and Amazon Redshift.
• Score: 18.665946271507117
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Database research and development often require a large number of SQL queries for benchmarking purposes. However, acquiring real-world SQL queries is challenging due to privacy concerns, and existing SQL generation methods are limited in customization and in satisfying realistic constraints. To address this issue, we present SQLBarber, a system based on Large Language Models (LLMs) to generate customized and realistic SQL workloads. SQLBarber (i) eliminates the need for users to manually craft SQL templates in advance, while providing the flexibility to accept natural language specifications to constrain SQL templates, (ii) scales efficiently to generate large volumes of queries matching any user-defined cost distribution (e.g., cardinality and execution plan cost), and (iii) uses execution statistics from Amazon Redshift and Snowflake to derive SQL template specifications and query cost distributions that reflect real-world query characteristics. SQLBarber introduces (i) a declarative interface for users to effortlessly generate customized SQL templates, (ii) an LLM-powered pipeline augmented with a self-correction module that profiles, refines, and prunes SQL templates based on query costs, and (iii) a Bayesian Optimizer to efficiently explore different predicate values and identify a set of queries that satisfy the target cost distribution. We construct and open-source ten benchmarks of varying difficulty levels and target query cost distributions based on real-world statistics from Snowflake and Amazon Redshift. Extensive experiments on these benchmarks show that SQLBarber is the only system that can generate customized SQL templates. It reduces query generation time by one to three orders of magnitude, and significantly improves alignment with the target cost distribution, compared with existing methods.

Related papers

• XiYan-SQL: A Novel Multi-Generator Framework For Text-to-SQL [48.45491386478092]
  We present XiYan-, an innovative framework effectively generating and utilizing multiplesql candidates.<n>Overall, XiYan- achieves a new SOTA performance of 75.63% on the notable BIRD benchmark.<n>It also attains SOTA performance on the Spider test set with an accuracy of 89.65%.
  arXiv  Detail & Related papers  (2025-07-07T06:50:46Z)
• HI-SQL: Optimizing Text-to-SQL Systems through Dynamic Hint Integration [1.3927943269211591]
  Text-to-generation bridges the gap between natural language and databases, enabling users to query data without requiringsql expertise.<n>We propose HI-the, a pipeline that incorporates a novel hint generation mechanism utilizing historical query logs.<n>By analyzing prior queries, our method generates contextual hints that focus on handling the complexities of multi-table and nested operations.<n>Our approach significantly improves query accuracy of LLM-generated queries while ensuring efficiency in terms of calls and latency.
  arXiv  Detail & Related papers  (2025-06-11T12:07:55Z)
• Query and Conquer: Execution-Guided SQL Generation [2.07180164747172]
  We propose a novel approach for generating complex outputs that significantly improves accuracy in text-to- tasks.<n>Our method leverages execution results to select the most semantically consistent query from multiple candidates.
  arXiv  Detail & Related papers  (2025-03-31T17:43:36Z)
• STaR-SQL: Self-Taught Reasoner for Text-to-SQL [20.719165038519744]
  "chain-of-thought" rationales have proven effective for improving the performance of large language models on complex reasoning tasks.<n>Applying such techniques to structured tasks, such as text-to-driven, remains largely unexplored.<n>In this paper, we introduce Self-Taughter for text-to-driven (STaR-), a novel approach that reframes query generation as a reasoning process.<n> Experimental results on the challenging Spider benchmark demonstrate that STaR- significantly improves text-to-performance, achieving an execution accuracy of 86.6%.<n>These findings underscore the potential of reasoning-augmented training for
  arXiv  Detail & Related papers  (2025-02-19T08:58:44Z)
• MCTS-SQL: An Effective Framework for Text-to-SQL with Monte Carlo Tree Search [3.521199751827158]
  We present a novel approach to converting natural language queries into database queries.<n>We use Monte Carlo Tree Search (MCTS) and a self-refinement mechanism to enhance accuracy and reliability.<n> Experimental results show that MCTS-IDER achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-01-28T00:52:23Z)
• CHESS: Contextual Harnessing for Efficient SQL Synthesis [1.9506402593665235]
  We introduce CHESS, a framework for efficient and scalable text-to- queries. It comprises four specialized agents, each targeting one of the aforementioned challenges. Our framework offers features that adapt to various deployment constraints.
  arXiv  Detail & Related papers  (2024-05-27T01:54:16Z)
• SQLPrompt: In-Context Text-to-SQL with Minimal Labeled Data [54.69489315952524]
  "Prompt" is designed to improve the few-shot prompting capabilities of Text-to-LLMs. "Prompt" outperforms previous approaches for in-context learning with few labeled data by a large margin. We show that emphPrompt outperforms previous approaches for in-context learning with few labeled data by a large margin.
  arXiv  Detail & Related papers  (2023-11-06T05:24:06Z)
• JoinGym: An Efficient Query Optimization Environment for Reinforcement Learning [58.71541261221863]
  Join order selection (JOS) is the problem of ordering join operations to minimize total query execution cost. We present JoinGym, a query optimization environment for bushy reinforcement learning (RL) Under the hood, JoinGym simulates a query plan's cost by looking up intermediate result cardinalities from a pre-computed dataset.
  arXiv  Detail & Related papers  (2023-07-21T17:00:06Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• Wav2SQL: Direct Generalizable Speech-To-SQL Parsing [55.10009651476589]
  Speech-to-Spider (S2Spider) aims to convert spoken questions intosql queries given databases. We propose the first direct speech-to-speaker parsing model Wav2 which avoids error compounding across cascaded systems. Experimental results demonstrate that Wav2 avoids error compounding and achieves state-of-the-art results by up to 2.5% accuracy improvement over the baseline.
  arXiv  Detail & Related papers  (2023-05-21T19:26:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.