Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity

• URL: http://arxiv.org/abs/2601.15728v2
• Date: Fri, 23 Jan 2026 06:25:56 GMT
• Title: Benchmarking Text-to-Python against Text-to-SQL: The Impact of Explicit Logic and Ambiguity
• Authors: Hangle Hu, Chenyu Hou, Bin Cao, Ruizhe Li,
• Abstract summary: We introduce BIRD-Python, a benchmark designed for cross-paradigm evaluation.<n>We show that Python requires explicit procedural logic, making it highly sensitive to under user intent.<n>We propose the Logic Completion Framework (LCF), which resolves ambiguity by incorporating latent domain knowledge into the generation process.
• Score: 5.794032059676749
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: While Text-to-SQL remains the dominant approach for database interaction, real-world analytics increasingly require the flexibility of general-purpose programming languages such as Python or Pandas to manage file-based data and complex analytical workflows. Despite this growing need, the reliability of Text-to-Python in core data retrieval remains underexplored relative to the mature SQL ecosystem. To address this gap, we introduce BIRD-Python, a benchmark designed for cross-paradigm evaluation. We systematically refined the original dataset to reduce annotation noise and align execution semantics, thereby establishing a consistent and standardized baseline for comparison. Our analysis reveals a fundamental paradigmatic divergence: whereas SQL leverages implicit DBMS behaviors through its declarative structure, Python requires explicit procedural logic, making it highly sensitive to underspecified user intent. To mitigate this challenge, we propose the Logic Completion Framework (LCF), which resolves ambiguity by incorporating latent domain knowledge into the generation process. Experimental results show that (1) performance differences primarily stem from missing domain context rather than inherent limitations in code generation, and (2) when these gaps are addressed, Text-to-Python achieves performance parity with Text-to-SQL. These findings establish Python as a viable foundation for analytical agents-provided that systems effectively ground ambiguous natural language inputs in executable logical specifications. Resources are available at https://anonymous.4open.science/r/Bird-Python-43B7/.

Related papers

• APEX-SQL: Talking to the data via Agentic Exploration for Text-to-SQL [39.76924093980244]
  APEX- verbalize is a framework that shifts the paradigm from passive translation to agentic exploration.<n>Our framework employs a hypothesis-verification loop to ground model reasoning in real data.
  arXiv  Detail & Related papers  (2026-02-11T07:50:47Z)
• Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL [30.78817492504152]
  HERO is a hierarchical representation approach that integrates global intent and local details.<n>We employ a Nested Message Passing Neural Network (NMPNN) to capture inherent information in relational schema-guided semantics.<n>Our approach outperforms existing state-of-the-art methods, achieving an average 9.40% improvement of AUPRC and 12.35% of AUROC in identifying semantic inconsistencies.<n>It excels at detecting fine-grained semantic errors, provides large language models with more granular feedback, and ultimately enhances the reliability and interpretability of data querying platforms.
  arXiv  Detail & Related papers  (2025-12-28T02:25:33Z)
• Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation [54.53145282349042]
  We introduce DSR-sourced, a textbfDual-textbfS textbfReasoning framework that models Text-to-context as an interaction between an adaptive context state and a progressive generation state.<n>Without any post-training or in-context examples, DSR-sourced achieves competitive performance, reaching 35.28% execution accuracy on Spider 2.0-Snow and 68.32% on BIRD development set.
  arXiv  Detail & Related papers  (2025-11-26T13:52:50Z)
• STARQA: A Question Answering Dataset for Complex Analytical Reasoning over Structured Databases [27.66819120859756]
  We introduce STARQA, the first public human-created dataset of complex analytical reasoning questions and answers on three specialized relational-domain databases.<n>In this paper, we introduce STARQA, the first public human-created dataset of complex analytical reasoning questions and answers on three specialized relational-domain databases.
  arXiv  Detail & Related papers  (2025-09-23T19:26:16Z)
• Text2VectorSQL: Towards a Unified Interface for Vector Search and SQL Queries [36.92547259037192]
  The proliferation of unstructured data poses a fundamental challenge to traditional database infrastructure.<n>While Text-to-BIRD has democratized access to structured data, it remains incapable of interpreting semantic or multi-modal queries.<n>We introduce and formalize Text2 Vector, a novel task to establish a unified natural language for seamlessly querying both structured and unstructured data.
  arXiv  Detail & Related papers  (2025-06-29T03:17:42Z)
• LogicCat: A Chain-of-Thought Text-to-SQL Benchmark for Complex Reasoning [12.249447967086828]
  LogicCat is the first Text-to-sense benchmark dataset specifically designed for complex reasoning and chain-of-thought parsing.<n>We show that LogicCat substantially increases the task difficulty for current state-of-the-art models to 33.20% execution accuracy.
  arXiv  Detail & Related papers  (2025-05-24T15:23:43Z)
• Enhancing Text-to-SQL Translation for Financial System Design [5.248014305403357]
  We consider Large Language Models (LLMs), which have achieved state of the art for various NLP tasks. We propose two novel metrics that were designed to adequately measure the similarity between relational queries.
  arXiv  Detail & Related papers  (2023-12-22T14:34:19Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL [49.85635994436742]
  We include human in the loop and present a novel-independent interactive approach (PIIA) that interacts with users using multi-choice questions. PIIA is capable of enhancing the text-to-domain performance with limited interaction turns by using both simulation and human evaluation.
  arXiv  Detail & Related papers  (2020-11-09T02:14:33Z)
• Photon: A Robust Cross-Domain Text-to-SQL System [189.1405317853752]
  We present Photon, a robust, modular, cross-domain NLIDB that can flag natural language input to which a mapping cannot be immediately determined. The proposed method effectively improves the robustness of text-to-native system against untranslatable user input.
  arXiv  Detail & Related papers  (2020-07-30T07:44:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.