CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems

• URL: http://arxiv.org/abs/2604.22313v1
• Date: Fri, 24 Apr 2026 07:47:16 GMT
• Title: CLARITY: A Framework and Benchmark for Conversational Language Ambiguity and Unanswerability in Interactive NL2SQL Systems
• Abstract summary: Existing benchmarks typically assume a single source of ambiguity and rely on user interaction for resolution.<n>We introduce Clarity, a framework for automatically generating an NL2 benchmark with multi-faceted ambiguities and diverse user behaviors.<n>Our results highlight the need for more robust ambiguity detection and resolution in industry-grade NL2 systems.
• Score: 13.595364881493943
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: NL2SQL systems deployed in industry settings often encounter ambiguous or unanswerable queries, particularly in interactive scenarios with incomplete user clarification. Existing benchmarks typically assume a single source of ambiguity and rely on user interaction for resolution, overlooking realistic failure modes. We introduce Clarity, a framework for automatically generating an NL2SQL benchmark with multi-faceted ambiguities and diverse user behaviors across both single- and multi-turn settings. Using a constraint-driven pipeline, Clarity transforms executable SQL into ambiguous queries, augmented with grounded conversational continuations and schema-level metadata. Empirical evaluation on Spider and BIRD shows that leading NL2SQL systems, including those based on strong LLMs, suffer significant performance degradation under multi-faceted ambiguity. While these systems often detect ambiguity, they struggle to accurately localize and resolve the underlying schema-level sources. Our results highlight the need for more robust ambiguity detection and resolution in industry-grade NL2SQL systems.

Related papers

• Reflect-SQL: A Self-Reflection Based Framework for Text-to-SQL [3.052889399420755]
  Democratizing data access through natural language is a crucial goal for modern enterprises.<n>The practical adoption of Text-to-effective is critically hindered by real-world complexities.<n>We introduce Reflect-, a novel framework for Text tosql, grounded in multi-stage self-reflection approach.<n>Our framework achieves an execution accuracy of 72.03%, significantly outperforming state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-09-01T07:46:27Z)
• SOMA-SQL: Resolving Multi-Source Ambiguity in NL-to-SQL via Synthetic Log and Execution Probing [37.513757157512686]
  Ambiguity across user questions, database schemas, and model interpretations are central failure modes in NL2.<n>We propose SOMA- to automatically resolve ambiguity via targeted synthetic query log and ambiguity-driven probing.<n> Experiments on six public benchmarks demonstrate that SOMA- improves execution accuracy by 13.0% on average over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2026-06-09T20:18:22Z)
• ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL [51.98164069124653]
  ProSPy is a Profiling-driven framework for enterprise-scale Text-to-gnostic analysis.<n>ProSPy structures the reasoning process into four stages.<n>It first extracts fine-grained data evidence through automatic profiling.<n>It prunes large schemas into task-relevant contexts.
  arXiv  Detail & Related papers  (2026-06-04T08:13:05Z)
• FD-NL2SQL: Feedback-Driven Clinical NL2SQL that Improves with Use [10.080176148010247]
  Clinicians exploring oncology trial repositories often need ad-hoc, multi-constraint queries over biomarkers, endpoints, interventions, and time.<n>We demo FD-NL2 synthesis, a feedback-driven exemplar interface for exemplar-based oncology databases.
  arXiv  Detail & Related papers  (2026-04-17T02:45:57Z)
• ErrorLLM: Modeling SQL Errors for Text-to-SQL Refinement [57.98138819417949]
  We propose ErrorLLM, a framework that explicitly models text-to- querying.<n>We show that ErrorLLM achieves the most significant improvements over backbone initial generation.<n>ErrorLLM addresses both sides by high detection F1 score while maintaining refinement effectiveness.
  arXiv  Detail & Related papers  (2026-03-04T05:27:20Z)
• Bridging Global Intent with Local Details: A Hierarchical Representation Approach for Semantic Validation in Text-to-SQL [30.78817492504152]
  HERO is a hierarchical representation approach that integrates global intent and local details.<n>We employ a Nested Message Passing Neural Network (NMPNN) to capture inherent information in relational schema-guided semantics.<n>Our approach outperforms existing state-of-the-art methods, achieving an average 9.40% improvement of AUPRC and 12.35% of AUROC in identifying semantic inconsistencies.<n>It excels at detecting fine-grained semantic errors, provides large language models with more granular feedback, and ultimately enhances the reliability and interpretability of data querying platforms.
  arXiv  Detail & Related papers  (2025-12-28T02:25:33Z)
• Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation [54.53145282349042]
  We introduce DSR-sourced, a textbfDual-textbfS textbfReasoning framework that models Text-to-context as an interaction between an adaptive context state and a progressive generation state.<n>Without any post-training or in-context examples, DSR-sourced achieves competitive performance, reaching 35.28% execution accuracy on Spider 2.0-Snow and 68.32% on BIRD development set.
  arXiv  Detail & Related papers  (2025-11-26T13:52:50Z)
• AmbiSQL: Interactive Ambiguity Detection and Resolution for Text-to-SQL [0.9217021281095907]
  We demonstrate Ambi, an interactive system that automatically detects query ambiguities and guides users through multiple-choice questions to clarify their intent.<n>Ambi achieves 87.2% in ambiguity detection and improves exact accuracy by 50% when integrated with Text-to- dataset systems.
  arXiv  Detail & Related papers  (2025-08-21T06:10:28Z)
• ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration [32.83579488224367]
  We present ReFoRCE, a Text-to-confidence agent that tops the Spider 2.0 leaderboard.<n>ReFoRCE achieves state-of-the-art results, with scores of 35.83 Spider 2.0-Snow and 36.56 on Spider 2.0-Lite.
  arXiv  Detail & Related papers  (2025-02-02T05:25:03Z)
• AMBROSIA: A Benchmark for Parsing Ambiguous Questions into Database Queries [56.82807063333088]
  We introduce a new benchmark, AMBROSIA, which we hope will inform and inspire the development of text-to-open programs. Our dataset contains questions showcasing three different types of ambiguity (scope ambiguity, attachment ambiguity, and vagueness) In each case, the ambiguity persists even when the database context is provided. This is achieved through a novel approach that involves controlled generation of databases from scratch.
  arXiv  Detail & Related papers  (2024-06-27T10:43:04Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.