Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration

• URL: http://arxiv.org/abs/2510.26495v1
• Date: Thu, 30 Oct 2025 13:44:22 GMT
• Title: Rethinking Text-to-SQL: Dynamic Multi-turn SQL Interaction for Real-world Database Exploration
• Authors: Linzhuang Sun, Tianyu Guo, Hao Liang, Yuying Li, Qifeng Cai, Jingxuan Wei, Bihui Yu, Wentao Zhang, Bin Cui,
• Abstract summary: We introduce Dy-Bench, a benchmark assessing model performance under evolving user interactions.<n>Unlike previous manually curated datasets, Dylz-Bench is built through an automated two-stage pipeline of task and verification.<n>Dy-Bench covers 13 domains across BIRD and Spider 2 databases, totaling 1,072 tasks.
• Score: 21.94739453628141
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advances in Text-to-SQL have achieved strong results in static, single-turn tasks, where models generate SQL queries from natural language questions. However, these systems fall short in real-world interactive scenarios, where user intents evolve and queries must be refined over multiple turns. In applications such as finance and business analytics, users iteratively adjust query constraints or dimensions based on intermediate results. To evaluate such dynamic capabilities, we introduce DySQL-Bench, a benchmark assessing model performance under evolving user interactions. Unlike previous manually curated datasets, DySQL-Bench is built through an automated two-stage pipeline of task synthesis and verification. Structured tree representations derived from raw database tables guide LLM-based task generation, followed by interaction-oriented filtering and expert validation. Human evaluation confirms 100% correctness of the synthesized data. We further propose a multi-turn evaluation framework simulating realistic interactions among an LLM-simulated user, the model under test, and an executable database. The model must adapt its reasoning and SQL generation as user intents change. DySQL-Bench covers 13 domains across BIRD and Spider 2 databases, totaling 1,072 tasks. Even GPT-4o attains only 58.34% overall accuracy and 23.81% on the Pass@5 metric, underscoring the benchmark's difficulty. All code and data are released at https://github.com/Aurora-slz/Real-World-SQL-Bench .

Related papers

• Monte Carlo Tree Search with Reasoning Path Refinement for Small Language Models in Conversational Text-to-NoSQL [20.156191782890797]
  We introduce the Conversational Text-to-No task, which generates queries given a natural language question, a database, and a dialogue history.<n>We propose Stage-MCTS, a framework that endows small language models with query-specific reasoning capabilities.<n>Our approach outperforms state-of-the-art large reasoning models, improving execution value match accuracy by up to 7.93%.
  arXiv  Detail & Related papers  (2026-02-13T03:35:38Z)
• LLM-Based SQL Generation: Prompting, Self-Refinement, and Adaptive Weighted Majority Voting [7.590911146338215]
  We propose a Single-Agent Self-Refinement with Ensemble Voting (SSEV)<n>We build on insights from the SSEV pipeline to address the growing complexity of enterprise databases and real-world Text-to-Act tasks.<n>ReCAPAgent-5.5% integrates specialized agents for planning, external knowledge retrieval, critique, action generation, self-refinement, schema linking, and result validation.
  arXiv  Detail & Related papers  (2026-01-25T18:38:58Z)
• Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation [54.53145282349042]
  We introduce DSR-sourced, a textbfDual-textbfS textbfReasoning framework that models Text-to-context as an interaction between an adaptive context state and a progressive generation state.<n>Without any post-training or in-context examples, DSR-sourced achieves competitive performance, reaching 35.28% execution accuracy on Spider 2.0-Snow and 68.32% on BIRD development set.
  arXiv  Detail & Related papers  (2025-11-26T13:52:50Z)
• RAISE: Reasoning Agent for Interactive SQL Exploration [47.77323087050061]
  We propose a novel framework that unifies schema linking, query generation, and iterative refinement within a single, end-to-end component.<n>Our method emulates how humans answer questions when working with unfamiliar databases.
  arXiv  Detail & Related papers  (2025-06-02T03:07:08Z)
• Bridging the Gap: Enabling Natural Language Queries for NoSQL Databases through Text-to-NoSQL Translation [25.638927795540454]
  We introduce the Text-to-No task, which aims to convert natural language queries into accessible queries.<n>To promote research in this area, we released a large-scale and open-source dataset for this task, named TEND (short interfaces for Text-to-No dataset)<n>We also designed a SLM (Small Language Model)-assisted and RAG (Retrieval-augmented Generation)-assisted multi-step framework called SMART, which is specifically designed for Text-to-No conversion.
  arXiv  Detail & Related papers  (2025-02-16T17:01:48Z)
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows [64.94146689665628]
  Spider 2.0 is an evaluation framework for real-world text-to-sql problems derived from enterprise-level database use cases.<n>The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake.<n>We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-levels.
  arXiv  Detail & Related papers  (2024-11-12T12:52:17Z)
• TrustSQL: Benchmarking Text-to-SQL Reliability with Penalty-Based Scoring [11.78795632771211]
  We introduce a novel benchmark designed to evaluate text-to- reliability as a model's ability to correctly handle any type of input question. We evaluate existing methods using a novel penalty-based scoring metric with two modeling approaches.
  arXiv  Detail & Related papers  (2024-03-23T16:12:52Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• Augmenting Multi-Turn Text-to-SQL Datasets with Self-Play [46.07002748587857]
  We explore augmenting the training datasets using self-play, which leverages contextual information to synthesize new interactions. We find that self-play improves the accuracy of a strong baseline on SParC and Co, two widely used text-to-domain datasets.
  arXiv  Detail & Related papers  (2022-10-21T16:40:07Z)
• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL [49.85635994436742]
  We include human in the loop and present a novel-independent interactive approach (PIIA) that interacts with users using multi-choice questions. PIIA is capable of enhancing the text-to-domain performance with limited interaction turns by using both simulation and human evaluation.
  arXiv  Detail & Related papers  (2020-11-09T02:14:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.