Benchmarking Text-to-SQL under Role-Based Access Control

• URL: http://arxiv.org/abs/2607.22115v1
• Date: Fri, 24 Jul 2026 09:10:58 GMT
• Title: Benchmarking Text-to-SQL under Role-Based Access Control
• Abstract summary: Currently, popular text-to- benchmarks mostly assume unrestricted access to S.<n>In practice, user access is often restricted, e.g., through role-based access control (RBAC) policies.<n>We present a comprehensive text-to-assisted RBAC framework that augments existing text-to- benchmarks with realistic user roles and access policies.
• Score: 27.035834672528548
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Given a database S and a natural language question Q, text-to-SQL systems aim to generate an SQL query that correctly answers Q when executed against S. Currently, popular text-to-SQL benchmarks mostly assume unrestricted access to S; in practice, however, user access is often restricted, e.g., through role-based access control (RBAC) policies. This leads to a potential disconnect between benchmarking results and real-world performance: an LLM with high benchmark scores might perform poorly in an access-controlled environment, by frequently violating RBAC, or rejecting a query q that could be answered with only permitted data in S. Motivated by this, we present a comprehensive text-to-SQL benchmarking framework with realistic RBAC constraints, which features an LLM-assisted workflow that augments existing text-to-SQL benchmarks with plausible user roles and access policies. To do so, we formulate the problem of role synthesis as a structured reasoning process over the database schema, in which the LLM first infers the application context from the schema, and then derives role responsibilities and access scopes consistent with this context. This process is audited by human-in-the-loop quality control, in which domain experts perform metric-guided screening on the generated roles. Besides the augmented dataset, the proposed framework also contains evaluation metrics that identify RBAC-specific failure modes, and disentangle SQL utility from access-control compliance. We apply the proposed framework to several widely-used benchmarks, and conduct a systematic empirical study of state-of-the-art text-to-SQL systems. The results show that many solutions (especially open-weight LLMs) with high benchmarking scores under an unrestricted setting suffer sharp performance degradation once access constraints are in place, due to frequent RBAC violations.

Related papers

• SafeQL: Search-based Refinement for Safe and Efficient LLM-based Text-to-SQL [4.9445294133117255]
  This paper proposes SafeQL, a search-based refinement paradigm that redefines the role of the interfaces as an active guide in the refinement process.<n> Experiments on the Bird and Spider benchmarks show that SafeQL significantly improves execution accuracy and efficiency compared to regeneration-based methods.
  arXiv  Detail & Related papers  (2026-08-10T08:16:17Z)
• SpotIt+: Verification-based Text-to-SQL Evaluation with Database Constraints [9.733987594033907]
  SpotIt+ is a tool for evaluating text-to-speech systems via bounded equivalence verification.<n>We introduce a constraint-mining pipeline that combines rule-based specification mining over example databases with LLM-based validation.<n> Experimental results on the BIRD dataset show that the mined constraints enable SpotIt+ to generate more realistic differentiating databases.
  arXiv  Detail & Related papers  (2026-03-04T17:51:42Z)
• ARBITER: AI-Driven Filtering for Role-Based Access Control [0.2519906683279152]
  our implements layered input/output validation, role-aware retrieval, and post-generation fact-checking.<n>We evaluate the approach on 389 queries using a synthetic dataset.<n>Results suggest that practical RBAC deployment on RAG systems is approaching the maturity level needed for dynamic enterprise environments.
  arXiv  Detail & Related papers  (2025-12-23T17:25:51Z)
• KBQA-R1: Reinforcing Large Language Models for Knowledge Base Question Answering [64.62317305868264]
  We present textbfKBQA-R1, a framework that shifts the paradigm from text imitation to interaction optimization via Reinforcement Learning.<n>Treating KBQA as a multi-turn decision process, our model learns to navigate the knowledge base using a list of actions.<n>Experiments on WebQSP, GrailQA, and GraphQuestions demonstrate that KBQA-R1 achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2025-12-10T17:45:42Z)
• LLM-Symbolic Integration for Robust Temporal Tabular Reasoning [69.27153114778748]
  We introduce TempTabQA-C, a synthetic dataset designed for systematic and controlled evaluations.<n>This structured approach allows Large Language Models (LLMs) to generate and executesql queries, enhancing generalization and mitigating biases.
  arXiv  Detail & Related papers  (2025-06-06T05:14:04Z)
• Text-to-SQL based on Large Language Models and Database Keyword Search [0.0]
  This paper proposes a strategy to compile Natural Language (NL) questions intosql queries.<n>The strategy incorporates a dynamic few-shot examples strategy and leverages the services provided by a database keyword search (KwS) platform.<n>Experiments show that the strategy achieves an accuracy on the real-world relational database that surpasses state-of-the-art approaches.
  arXiv  Detail & Related papers  (2025-01-23T12:03:29Z)
• Lucy: Think and Reason to Solve Text-to-SQL [12.52968634440807]
  Large Language Models (LLMs) have made significant progress in assisting users to query databases in natural language. LLMs provide state-of-the-art results on many standard benchmarks, but their performance significantly drops when applied to large enterprise databases. We propose a new solution that combines the power of LLMs in understanding questions with automated reasoning techniques to handle complex database constraints.
  arXiv  Detail & Related papers  (2024-07-06T18:56:42Z)
• Can Long-Context Language Models Subsume Retrieval, RAG, SQL, and More? [54.667202878390526]
  Long-context language models (LCLMs) have the potential to revolutionize our approach to tasks traditionally reliant on external tools like retrieval systems or databases. We introduce LOFT, a benchmark of real-world tasks requiring context up to millions of tokens designed to evaluate LCLMs' performance on in-context retrieval and reasoning. Our findings reveal LCLMs' surprising ability to rival state-of-the-art retrieval and RAG systems, despite never having been explicitly trained for these tasks.
  arXiv  Detail & Related papers  (2024-06-19T00:28:58Z)
• DePLOI: Applying NL2SQL to Synthesize and Audit Database Access Control [6.2859996652179]
  This paper introduces a new access control model called Intent-Based Access Control for Databases (IBAC-DB)<n>In IBAC-DB, access control policies are expressed using abstractions that scale to high numbers of database objects, and are traceable with respect to implementations.<n>This paper proposes DePLOI, a system leveraging access control-specific task decompositions to accurately synthesize and audit access control implementation from IBAC-DB abstractions.
  arXiv  Detail & Related papers  (2024-02-11T23:50:12Z)
• CRUD-RAG: A Comprehensive Chinese Benchmark for Retrieval-Augmented Generation of Large Language Models [49.16989035566899]
  Retrieval-Augmented Generation (RAG) is a technique that enhances the capabilities of large language models (LLMs) by incorporating external knowledge sources. This paper constructs a large-scale and more comprehensive benchmark, and evaluates all the components of RAG systems in various RAG application scenarios.
  arXiv  Detail & Related papers  (2024-01-30T14:25:32Z)
• Text-to-SQL Empowered by Large Language Models: A Benchmark Evaluation [76.76046657162306]
  Large language models (LLMs) have emerged as a new paradigm for Text-to- task. Large language models (LLMs) have emerged as a new paradigm for Text-to- task.
  arXiv  Detail & Related papers  (2023-08-29T14:59:54Z)
• Allies: Prompting Large Language Model with Beam Search [107.38790111856761]
  In this work, we propose a novel method called ALLIES. Given an input query, ALLIES leverages LLMs to iteratively generate new queries related to the original query. By iteratively refining and expanding the scope of the original query, ALLIES captures and utilizes hidden knowledge that may not be directly through retrieval.
  arXiv  Detail & Related papers  (2023-05-24T06:16:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.