ScenarioBench: Trace-Grounded Compliance Evaluation for Text-to-SQL and RAG

• URL: http://arxiv.org/abs/2509.24212v1
• Date: Mon, 29 Sep 2025 02:51:08 GMT
• Title: ScenarioBench: Trace-Grounded Compliance Evaluation for Text-to-SQL and RAG
• Abstract summary: Scenario is a policy-grounded, trace-aware benchmark for evaluating Text-to-and-retrieval-augmented generation in compliance contexts.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: ScenarioBench is a policy-grounded, trace-aware benchmark for evaluating Text-to-SQL and retrieval-augmented generation in compliance contexts. Each YAML scenario includes a no-peek gold-standard package with the expected decision, a minimal witness trace, the governing clause set, and the canonical SQL, enabling end-to-end scoring of both what a system decides and why. Systems must justify outputs using clause IDs from the same policy canon, making explanations falsifiable and audit-ready. The evaluator reports decision accuracy, trace quality (completeness, correctness, order), retrieval effectiveness, SQL correctness via result-set equivalence, policy coverage, latency, and an explanation-hallucination rate. A normalized Scenario Difficulty Index (SDI) and a budgeted variant (SDI-R) aggregate results while accounting for retrieval difficulty and time. Compared with prior Text-to-SQL or KILT/RAG benchmarks, ScenarioBench ties each decision to clause-level evidence under strict grounding and no-peek rules, shifting gains toward justification quality under explicit time budgets.

Related papers

• HARP: Hierarchical Adaptive Ranking with Preference-Adaptive Fusion for Query-Based CVE Prioritization [84.9738509922482]
  We propose a graph-grounded multi-view framework that ranks candidates from a natural-language query together with a support bank of historical labeled examples.<n>HarP retrieves evidence from a vulnerability knowledge graph, scores candidates with policy-conditioned global, enterprise, and user views, and fits view-fusion weights from sampled supports.
  arXiv  Detail & Related papers  (2026-08-19T20:29:00Z)
• Benchmarking Text-to-SQL under Role-Based Access Control [27.035834672528548]
  Currently, popular text-to- benchmarks mostly assume unrestricted access to S.<n>In practice, user access is often restricted, e.g., through role-based access control (RBAC) policies.<n>We present a comprehensive text-to-assisted RBAC framework that augments existing text-to- benchmarks with realistic user roles and access policies.
  arXiv  Detail & Related papers  (2026-07-24T09:10:58Z)
• EASE-TTT: Evidence-Aligned Selective Test-Time Training for Long-Context Question Answering [61.89411578705886]
  We propose Evidence-Aligned SElective Test-Time Training (EASE-TTT) for long-context question answering.<n>EASE-TTT converts selected evidence chunks into a soft attention supervision target over their token positions.<n>It achieves the strongest macro-average performance among full-context inference, retrieval-only baselines, and qTTT.
  arXiv  Detail & Related papers  (2026-06-05T04:49:37Z)
• Agent-Agnostic Evaluation of SQL Accuracy in Production Text-to-SQL Systems [2.051598597332424]
  Text-to-native (T2native) evaluation in production environments poses fundamental challenges that existing benchmarks do not address.<n>We present STEF, a production- evaluation system that operates exclusively on natural language inputs.<n>STEF produces an interpretable 0 to 100 accuracy score via a composite metric that encompasses filter alignment, semantic verdict, and confidence of the evaluator.
  arXiv  Detail & Related papers  (2026-04-30T15:59:28Z)
• PV-SQL: Synergizing Database Probing and Rule-based Verification for Text-to-SQL Agents [7.865833770467325]
  Text-to- systems often struggle with deep contextual understanding, particularly for complex queries with subtle requirements.<n>We present PV-Verify, an agentic framework that addresses these failures through two complementary components: Probe and Verify.<n>The Probe component iteratively generates probing queries to retrieve concrete records from the database, resolving ambiguities in value formats, column semantics, and inter-table relationships to build richer contextual understanding.
  arXiv  Detail & Related papers  (2026-04-19T22:54:05Z)
• SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints [9.733987594033907]
  SpotIt+ is a tool for evaluating text-to-speech systems via bounded equivalence verification.<n>We introduce a constraint-mining pipeline that combines rule-based specification mining over example databases with LLM-based validation.<n> Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases.
  arXiv  Detail & Related papers  (2026-03-04T17:51:42Z)
• Companion Agents: A Table-Information Mining Paradigm for Text-to-SQL [8.159121916366727]
  Large-scale Text-to-curated benchmarks such as BIRD typically assume complete and accurate database annotations as well as available external knowledge.<n>This mismatch substantially limits the real-world applicability of state-of-the-domain Text-to-art systems.<n>We propose a database-centric approach that leverages intrinsic, fine-grained information residing in relational databases to construct missing evidence.
  arXiv  Detail & Related papers  (2025-12-17T07:11:55Z)
• Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation [54.53145282349042]
  We introduce DSR-sourced, a textbfDual-textbfS textbfReasoning framework that models Text-to-context as an interaction between an adaptive context state and a progressive generation state.<n>Without any post-training or in-context examples, DSR-sourced achieves competitive performance, reaching 35.28% execution accuracy on Spider 2.0-Snow and 68.32% on BIRD development set.
  arXiv  Detail & Related papers  (2025-11-26T13:52:50Z)
• SCARE: A Benchmark for SQL Correction and Question Answerability Classification for Reliable EHR Question Answering [18.161591137171623]
  We introduce SCARE, a benchmark for evaluating methods that function as a post-hoc safety layer in EHR QA systems.<n>SCARE evaluates the joint task of (1) classifying question answerability (i.e., determining whether a question is answerable, ambiguous, or unanswerable) and (2) verifying or correcting candidatesql queries.
  arXiv  Detail & Related papers  (2025-11-13T06:35:29Z)
• Learning to Route: A Rule-Driven Agent Framework for Hybrid-Source Retrieval-Augmented Generation [55.47971671635531]
  Large Language Models (LLMs) have shown remarkable performance on general Question Answering (QA)<n>Retrieval-Augmented Generation (RAG) addresses this limitation by enriching LLMs with external knowledge.<n>Existing systems primarily rely on unstructured documents, while largely overlooking relational databases.
  arXiv  Detail & Related papers  (2025-09-30T22:19:44Z)
• SQLCritic: Correcting Text-to-SQL Generation via Clause-wise Critic [8.680252929322684]
  We introduce a clause-wise critique generation task along with a benchmark,sqlCriticBench, which performs fine-grained error localization.<n>We also propose an automatically training dataset curation pipeline which annotates clause-wise critique at scale.
  arXiv  Detail & Related papers  (2025-03-11T02:52:39Z)
• PICASO: Permutation-Invariant Context Composition with State Space Models [98.91198288025117]
  State Space Models (SSMs) offer a promising solution by allowing a database of contexts to be mapped onto fixed-dimensional states.<n>We propose a simple mathematical relation derived from SSM dynamics to compose multiple states into one that efficiently approximates the effect of concatenating raw context tokens.<n>We evaluate our resulting method on WikiText and MSMARCO in both zero-shot and fine-tuned settings, and show that we can match the strongest performing baseline while enjoying on average 5.4x speedup.
  arXiv  Detail & Related papers  (2025-02-24T19:48:00Z)
• Fundamental Challenges in Evaluating Text2SQL Solutions and Detecting Their Limitations [9.363953429870007]
  We propose a unified taxonomy of all Text2 limitations that can lead to both prediction and evaluation errors.<n>We describe the causes of limitations with real-world examples and propose potential mitigation solutions for each category in the taxonomy.
  arXiv  Detail & Related papers  (2025-01-30T08:31:09Z)
• Few-shot Policy (de)composition in Conversational Question Answering [54.259440408606515]
  We propose a neuro-symbolic framework to detect policy compliance using large language models (LLMs) in a few-shot setting.<n>We show that our approach soundly reasons about policy compliance conversations by extracting sub-questions to be answered, assigning truth values from contextual information, and explicitly producing a set of logic statements from the given policies.<n>We apply this approach to the popular PCD and conversational machine reading benchmark, ShARC, and show competitive performance with no task-specific finetuning.
  arXiv  Detail & Related papers  (2025-01-20T08:40:15Z)
• Evaluating LLMs for Text-to-SQL Generation With Complex SQL Workload [1.2738020945091273]
  TPC-DS queries exhibit a significantly higher level of structural complexity compared to the other two benchmarks. Findings indicate that the current state-of-the-art generative AI models fall short in generating accurate decision-making queries. Results demonstrated that the accuracy of the generated queries is insufficient for practical real-world application.
  arXiv  Detail & Related papers  (2024-07-28T15:53:05Z)
• SWING: Balancing Coverage and Faithfulness for Dialogue Summarization [67.76393867114923]
  We propose to utilize natural language inference (NLI) models to improve coverage while avoiding factual inconsistencies. We use NLI to compute fine-grained training signals to encourage the model to generate content in the reference summaries that have not been covered. Experiments on the DialogSum and SAMSum datasets confirm the effectiveness of the proposed approach.
  arXiv  Detail & Related papers  (2023-01-25T09:33:11Z)
• STAR: SQL Guided Pre-Training for Context-dependent Text-to-SQL Parsing [64.80483736666123]
  We propose a novel pre-training framework STAR for context-dependent text-to- parsing. In addition, we construct a large-scale context-dependent text-to-the-art conversation corpus to pre-train STAR. Extensive experiments show that STAR achieves new state-of-the-art performance on two downstream benchmarks.
  arXiv  Detail & Related papers  (2022-10-21T11:30:07Z)
• Searching for Better Database Queries in the Outputs of Semantic Parsers [16.221439565760058]
  In this paper, we consider the case when, at the test time, the system has access to an external criterion that evaluates the generated queries. The criterion can vary from checking that a query executes without errors to verifying the query on a set of tests. We apply our approach to the state-of-the-art semantics and report that it allows us to find many queries passing all the tests on different datasets.
  arXiv  Detail & Related papers  (2022-10-13T17:20:45Z)
• SUN: Exploring Intrinsic Uncertainties in Text-to-SQL Parsers [61.48159785138462]
  This paper aims to improve the performance of text-to-dependence by exploring the intrinsic uncertainties in the neural network based approaches (called SUN) Extensive experiments on five benchmark datasets demonstrate that our method significantly outperforms competitors and achieves new state-of-the-art results.
  arXiv  Detail & Related papers  (2022-09-14T06:27:51Z)
• Towards Dynamic Consistency Checking in Goal-directed Predicate Answer Set Programming [2.3204178451683264]
  We present a variation of the top-down evaluation strategy, termed Dynamic Consistency checking. This makes it possible to determine when a literal is not compatible with the denials associated to the global constraints in the program. We have experimentally observed speedups of up to 90x w.r.t. the standard versions of s(CASP)
  arXiv  Detail & Related papers  (2021-10-22T20:38:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.