TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification

• URL: http://arxiv.org/abs/2608.17795v2
• Date: Mon, 24 Aug 2026 06:50:02 GMT
• Title: TraceSQL: Traceable Answerability Estimation for Reference-Free Text-to-SQL Verification
• Abstract summary: Given only a user question, database context, and generatedsql, can a system estimate whether the generated query is likely to correctly answer the question?<n>Recent approaches use LLMs as judge or specialized agents to inspect generatedsql, but their decisions can be difficult to trace.<n>We propose Trace, a lightweight and traceable verification model built on explicit diagnostic features.<n>Trace achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 5826% ROC-AUC for the Grade-7B ORM baseline on the same generated- semantic evaluation.
• Score: 0.14658400971135652
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-SQL systems are commonly evaluated using ground-truth SQL queries or reference execution results, but such supervision is unavailable at inference time in real-world deployments. This creates a critical verification problem: given only a user question, database context, and generated SQL, can a system estimate whether the generated query is likely to correctly answer the question? Recent approaches use LLMs as judge or specialized agents to inspect generated SQL, but their decisions can be difficult to trace. Outcome Reward Models (ORMs) address this by learning from execution-labeled candidate SQLs and assigning correctness scores to unseen queries, yet they still provide limited visibility into the signals behind each verification. To address this limitation, we propose TraceSQL, a lightweight and traceable verification model built on explicit diagnostic features. TraceSQL combines 67 features capturing question ambiguity, question requirements, question-schema-SQL consistency, SQL structure, and intent alignment. These signals remain available for examining which factors influence each prediction and for tracing decisions back to diagnostic evidence. On BIRD development databases, TraceSQL achieves 66.47% F1 and 64.48% ROC-AUC, compared with 61.87% F1 and 58.26% ROC-AUC for the GradeSQL-7B ORM baseline on the same generated-SQL evaluation. Feature attribution further shows that the model relies on both semantic grounding and deterministic SQL-structure signals. These results show that SQL verification can be performed with a lightweight learned model while retaining feature-level evidence for inspecting and diagnosing its predictions.

Related papers

• Spider 2.0-AIFunc: Extending Real-World Text-to-SQL to AI-Native SQL Workflows [54.49251107006261]
  We introduce Spider 2.0-AIFunc, a benchmark of 465 verified instances across 125 real-world databases covering six types of AI functions on the Snowflake platform.<n>We find that the strongest proprietary models reach 67-70% execution accuracy while the best open-source model achieves 58.1%.
  arXiv  Detail & Related papers  (2026-07-07T12:54:08Z)
• ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL [51.98164069124653]
  ProSPy is a Profiling-driven framework for enterprise-scale Text-to-gnostic analysis.<n>ProSPy structures the reasoning process into four stages.<n>It first extracts fine-grained data evidence through automatic profiling.<n>It prunes large schemas into task-relevant contexts.
  arXiv  Detail & Related papers  (2026-06-04T08:13:05Z)
• CAPER: Clause-Aligned Process Supervision for Text-to-SQL [11.286563956844946]
  We propose CAPER, which automatically derives clause-level supervision via counterfactual intervention on the SQL abstract syntax tree.<n>The resulting data is used to train CAPER-9B, a lightweight Clause-boundary feedback for policy optimization and candidate verification.<n> Experiments on BIRD and Spider show that clause-aligned supervision not only improves execution accuracy, achieving up to a 15.3% relative EX improvement over GPT-5.4, but also strengthens failure-localization capability, reaching 84.53% accuracy and 90.60% MRR on held-out failures.
  arXiv  Detail & Related papers  (2026-06-02T08:35:40Z)
• ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement [57.98138819417949]
  We propose ErrorLLM, a framework that explicitly models text-to- querying.<n>We show that ErrorLLM achieves the most significant improvements over backbone initial generation.<n>ErrorLLM addresses both sides by high detection F1 score while maintaining refinement effectiveness.
  arXiv  Detail & Related papers  (2026-03-04T05:27:20Z)
• Query Carefully: Detecting the Unanswerables in Text-to-SQL Tasks [1.7781743265224403]
  Text-to- systems allow non- experts to interact with databases using natural language.<n>Their tendency to generate executablesql for ambiguous, out-of-scope, or unanswerable queries introduces a hidden risk, as outputs may be misinterpreted as correct.<n>We present Query, a pipeline that integratessql generation with explicit ambiguity and handling of unanswerable inputs.
  arXiv  Detail & Related papers  (2025-12-19T12:22:27Z)
• STaR-SQL: Self-Taught Reasoner for Text-to-SQL [20.719165038519744]
  "chain-of-thought" rationales have proven effective for improving the performance of large language models on complex reasoning tasks.<n>Applying such techniques to structured tasks, such as text-to-driven, remains largely unexplored.<n>In this paper, we introduce Self-Taughter for text-to-driven (STaR-), a novel approach that reframes query generation as a reasoning process.<n> Experimental results on the challenging Spider benchmark demonstrate that STaR- significantly improves text-to-performance, achieving an execution accuracy of 86.6%.<n>These findings underscore the potential of reasoning-augmented training for
  arXiv  Detail & Related papers  (2025-02-19T08:58:44Z)
• Can the Rookies Cut the Tough Cookie? Exploring the Use of LLMs for SQL Equivalence Checking [15.42143912008553]
  We introduce a novel, realistic, and sufficiently complex benchmark called SQLEquiQuest for query equivalence checking.<n>We evaluate several state-of-the-art LLMs using various prompting strategies and carefully constructed in-context learning examples.<n>Our analysis shows that LLMs exhibit a strong bias for equivalence predictions, with consistently poor performance over non-equivalent pairs.
  arXiv  Detail & Related papers  (2024-12-07T06:50:12Z)
• RSL-SQL: Robust Schema Linking in Text-to-SQL Generation [51.00761167842468]
  We propose a novel framework called RSL- that combines bidirectional schema linking, contextual information augmentation, binary selection strategy, and multi-turn self-correction. benchmarks demonstrate that our approach achieves SOTA execution accuracy among open-source solutions, with 67.2% on BIRD and 87.9% on GPT-4ocorrection. Our approach outperforms a series of GPT-4 based Text-to-Seek systems when adopting DeepSeek (much cheaper) with same intact prompts.
  arXiv  Detail & Related papers  (2024-10-31T16:22:26Z)
• E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL [1.187832944550453]
  We introduce E-Seek, a novel pipeline specifically designed to address these challenges through direct schema linking and candidate predicate augmentation.<n>E-Seek enhances the natural language query by incorporating relevant database items (i.e., tables, columns, and values) and conditions directly into the question andsql construction plan, bridging the gap between the query and the database structure.<n> Comprehensive evaluations illustrate that E-Seek achieves competitive performance, particularly excelling in complex queries with a 66.29% execution accuracy on the test set.
  arXiv  Detail & Related papers  (2024-09-25T09:02:48Z)
• DAC: Decomposed Automation Correction for Text-to-SQL [51.48239006107272]
  We introduce De Automation Correction (DAC), which corrects text-to-composed by decomposing entity linking and skeleton parsing. We show that our method improves performance by $3.7%$ on average of Spider, Bird, and KaggleDBQA compared with the baseline method.
  arXiv  Detail & Related papers  (2024-08-16T14:43:15Z)
• Weakly Supervised Text-to-SQL Parsing through Question Decomposition [53.22128541030441]
  We take advantage of the recently proposed question meaning representation called QDMR. Given questions, their QDMR structures (annotated by non-experts or automatically predicted) and the answers, we are able to automatically synthesizesql queries. Our results show that the weakly supervised models perform competitively with those trained on NL- benchmark data.
  arXiv  Detail & Related papers  (2021-12-12T20:02:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.