BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations

• URL: http://arxiv.org/abs/2608.29345v1
• Date: Sat, 29 Aug 2026 15:59:56 GMT
• Title: BIRD-History: A Benchmark for History-Driven Text-to-SQL with Fine-Grained Knowledge Annotations
• Abstract summary: BIRD-History is a benchmark for evaluating text-to-sourced systems' ability to ground underspecified natural language questions.<n>We propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation.<n>Experiments demonstrate consistent improvements across four text-to-sourced systems.
• Score: 28.456885492014475
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While recent Large Language Model (LLM)-based text-to-SQL systems achieve impressive performance on standard benchmarks, they struggle when user queries implicitly rely on domain-specific knowledge, such as business logic, data conventions, and analytical practices, that is neither captured by the schema nor explicitly stated in the natural language question. Historical SQL query logs offer a valuable source of such knowledge, yet existing benchmarks do not adequately support evaluation of history-driven approaches. To address this gap, we introduce BIRD-History, a benchmark consisting of 1,393 tasks across 11 databases, designed to evaluate text-to-SQL systems' ability to ground underspecified natural language questions using historical SQL scripts. Each task is annotated with ground-truth labels specifying which historical queries contain relevant knowledge and which SQL clauses encode it, enabling systematic evaluation of both retrieval effectiveness and knowledge utilization. Alongside the benchmark, we propose a plug-in retriever that extracts five types of external knowledge from historical SQL scripts, then retrieves and reranks relevant fragments for query generation. The retriever integrates seamlessly into existing few-shot text-to-SQL pipelines without requiring prompt modifications. Experiments demonstrate consistent improvements across four text-to-SQL systems, highlighting the value of leveraging historical query logs for handling underspecified queries. Dataset and code are open-sourced on https://github.com/zjuidg/BIRD-History.

Related papers

• ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL [51.98164069124653]
  ProSPy is a Profiling-driven framework for enterprise-scale Text-to-gnostic analysis.<n>ProSPy structures the reasoning process into four stages.<n>It first extracts fine-grained data evidence through automatic profiling.<n>It prunes large schemas into task-relevant contexts.
  arXiv  Detail & Related papers  (2026-06-04T08:13:05Z)
• EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge [20.59995531311023]
  Existing benchmarks such as Spider, BIRD, and Spider2.0 evaluate schema generalization, large-scale databases, and realistic.<n>We introduce Ent, an enterprise-oriented Text-to-private benchmark for evaluating long-context grounding over proprietary business documents.<n>On English inputs, the best evaluated system reaches only 15.9% when long-form documents are provided.
  arXiv  Detail & Related papers  (2026-06-02T09:12:57Z)
• PARROT: A Benchmark for Evaluating LLMs in Cross-System SQL Translation [21.0303026118673]
  We introduce PARROT, a practical and realistic benchmak for CrOss-System SQL Translation.<n> PARROT comprises 598 translation pairs from 38 open-source benchmarks and real-world business services.<n>We also provide multiple benchmark variants, including PARROT-Diverse with 28,003 translations and PARROT-Simple with 5,306 representative samples.
  arXiv  Detail & Related papers  (2025-09-27T14:41:13Z)
• Datrics Text2SQL: A Framework for Natural Language to SQL Query Generation [0.0]
  This paper introduces a Retrieval-Augmented Generation (RAG)-based framework designed to generate accuratesql queries by leveraging structured documentation, example-based learning, and domain-specific rules.<n>The paper details the architecture, training methodology, and retrieval logic, highlighting how the system bridges the gap between user intent and database structure without requiringsql expertise.
  arXiv  Detail & Related papers  (2025-04-03T21:09:59Z)
• Text2SQL is Not Enough: Unifying AI and Databases with TAG [47.45480855418987]
  Table-Augmented Generation (TAG) is a paradigm for answering natural language questions over databases. We develop benchmarks to study the TAG problem and find that standard methods answer no more than 20% of queries correctly.
  arXiv  Detail & Related papers  (2024-08-27T00:50:14Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs [89.68522473384522]
  We present Bird, a big benchmark for large-scale database grounded in text-to-efficient tasks. Our emphasis on database values highlights the new challenges of dirty database contents. Even the most effective text-to-efficient models, i.e. ChatGPT, achieves only 40.08% in execution accuracy.
  arXiv  Detail & Related papers  (2023-05-04T19:02:29Z)
• A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions [102.8606542189429]
  The goal of text-to-corpora parsing is to convert a natural language (NL) question to its corresponding structured query language () based on the evidences provided by databases. Deep neural networks have significantly advanced this task by neural generation models, which automatically learn a mapping function from an input NL question to an output query.
  arXiv  Detail & Related papers  (2022-08-29T14:24:13Z)
• HIE-SQL: History Information Enhanced Network for Context-Dependent Text-to-SQL Semantic Parsing [1.343950231082215]
  We propose a History Information Enhanced text-to-the-art model (HIE-) to exploit context-dependence information from both history utterances and the last predictedsql query. We show our methods improve the performance of HIE- by a significant margin, which achieves new state-of-the-art results on the two context-dependent text-to-the-art benchmarks.
  arXiv  Detail & Related papers  (2022-03-14T11:58:37Z)
• Weakly Supervised Text-to-SQL Parsing through Question Decomposition [53.22128541030441]
  We take advantage of the recently proposed question meaning representation called QDMR. Given questions, their QDMR structures (annotated by non-experts or automatically predicted) and the answers, we are able to automatically synthesizesql queries. Our results show that the weakly supervised models perform competitively with those trained on NL- benchmark data.
  arXiv  Detail & Related papers  (2021-12-12T20:02:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.