Database Context Compression for Text-to-SQL on Real-World Large Databases

• URL: http://arxiv.org/abs/2606.28601v1
• Date: Fri, 26 Jun 2026 20:49:46 GMT
• Title: Database Context Compression for Text-to-SQL on Real-World Large Databases
• Abstract summary: We argue that the main bottleneck is no longer reasoning, but database representation.<n>Real databases contain repeated audit columns, large groups of similar tables, opaque identifiers whose meanings are stored only in documentation, and extensive data dictionaries with little query-relevant information.<n>We reformulate the problem as database context compression, a query-alinking transformation that rewrites schemas, semantic descriptions, and external documentation into a compact representation.
• Score: 23.20528827913883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent progress in Text-to-SQL has been driven by stronger language models and prompting strategies, yet performance on real enterprise benchmarks such as Spider 2.0 and BIRD remains far below that on classical academic datasets. We argue that the main bottleneck is no longer reasoning, but database representation. Real databases contain repeated audit columns, large groups of similar tables, opaque identifiers whose meanings are stored only in documentation, and extensive data dictionaries with little query-relevant information. Existing query-aware methods, including schema linking and retrieval-based schema selection, filter this raw context but still operate on redundant and verbose representations. We reformulate the problem as database context compression, a query-agnostic transformation that rewrites schemas, semantic descriptions, and external documentation into a compact representation. We formalize this transformation with the SGCF (Support-Gain Component Factorization) principle, which unifies repeated column extraction, isomorphic table templating, semantic componentization, and evidence purification under a single coverage objective. Based on SGCF, we propose DBCC, a database-side middleware that performs offline structural and semantic compression together with lightweight online evidence purification. DBCC is model-agnostic and can be integrated into existing Text-to-SQL pipelines. On Spider 2.0-Snow and BIRD, DBCC reduces input context by up to two orders of magnitude (from 2.6M to 34.7K tokens on the largest Spider 2.0-Snow subset), improves schema-linking strict recall from 0% to 56.5% under DeepSeek-V3.2 (63.1% under Claude Opus 4.7), and consistently increases end-to-end execution accuracy by 1.8-1.9% over three recent Text-to-SQL systems. Our code is open-sourced at https://github.com/MrBlankness/SchemaCompression.

Related papers

• A Semantic-Layer-Mediated Agent for Natural Language to SQL over Heterogeneous Enterprise Databases [0.0]
  We present a semantic-layer-mediated NL2 agent that decouples semantic intent from physicalsql execution.<n>The system employs a constrained think-act loop, supports BigQuery, and Snowflake backends, and is integrated into an end-to-end evaluation framework.<n>Using Gemini 3 Pro, the system 94.15% execution accuracy on the 547-task Spider2-snow benchmark, ranking third on the official leaderboard and substantially outperforming schema-only approaches.
  arXiv  Detail & Related papers  (2026-06-30T02:22:39Z)
• DecoSearch: Complexity-Aware Routing and Plan-Level Repair for Text-to-SQL [21.568773170643084]
  DecoSearch is a training-free framework that routing each query to the appropriate level of reasoning effort.<n>It achieves 70.53% execution accuracy on BIRD and 88.31% on Spider with a DeepSeek backbone.
  arXiv  Detail & Related papers  (2026-06-16T11:48:50Z)
• ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL [51.98164069124653]
  ProSPy is a Profiling-driven framework for enterprise-scale Text-to-gnostic analysis.<n>ProSPy structures the reasoning process into four stages.<n>It first extracts fine-grained data evidence through automatic profiling.<n>It prunes large schemas into task-relevant contexts.
  arXiv  Detail & Related papers  (2026-06-04T08:13:05Z)
• Text-to-SQL as Dual-State Reasoning: Integrating Adaptive Context and Progressive Generation [54.53145282349042]
  We introduce DSR-sourced, a textbfDual-textbfS textbfReasoning framework that models Text-to-context as an interaction between an adaptive context state and a progressive generation state.<n>Without any post-training or in-context examples, DSR-sourced achieves competitive performance, reaching 35.28% execution accuracy on Spider 2.0-Snow and 68.32% on BIRD development set.
  arXiv  Detail & Related papers  (2025-11-26T13:52:50Z)
• AutoLink: Autonomous Schema Exploration and Expansion for Scalable Schema Linking in Text-to-SQL at Scale [68.29002818359844]
  Existing methods incur prohibitive costs, struggle to trade off recall and noise, and scale poorly to large databases.<n>We present textbfAutoLink, an autonomous agent framework that reformulates schema linking as an iterative, agent-driven process.<n>Experiments demonstrate AutoLink's superior performance, achieving state-of-the-art strict schema linking recall, i.e., textbf68.7% EX on Bird-Dev (better than CHESS) and textbf34.9% EX on Spider-2.0-Lite (ranking 2nd
  arXiv  Detail & Related papers  (2025-11-21T12:12:17Z)
• UNJOIN: Enhancing Multi-Table Text-to-SQL Generation via Schema Simplification [50.59009084277447]
  We introduce UNJOIN, a framework that decouples the retrieval of schema elements from logic generation.<n>In the first stage, we merge the column names of all tables in the database into a single-table representation by prefixing each column with its table name.<n>In the second stage, the query is generated on this simplified schema and mapped back to the original schema by reconstructing JOINs, UNIONs, and relational logic.
  arXiv  Detail & Related papers  (2025-05-23T17:28:43Z)
• LinkAlign: Scalable Schema Linking for Real-World Large-Scale Multi-Database Text-to-SQL [24.740161980003652]
  LinkAlign is a novel framework tailored for large-scale databases with thousands of fields.<n>LinkAlign consistently outperforms existing baselines on all schema linking metrics.<n>It achieves a new state-of-the-art score of 33.09% on the Spider 2.0-Lite benchmark.
  arXiv  Detail & Related papers  (2025-03-24T11:53:06Z)
• ReFoRCE: A Text-to-SQL Agent with Self-Refinement, Consensus Enforcement, and Column Exploration [32.83579488224367]
  We present ReFoRCE, a Text-to-confidence agent that tops the Spider 2.0 leaderboard.<n>ReFoRCE achieves state-of-the-art results, with scores of 35.83 Spider 2.0-Snow and 36.56 on Spider 2.0-Lite.
  arXiv  Detail & Related papers  (2025-02-02T05:25:03Z)
• RSL-SQL: Robust Schema Linking in Text-to-SQL Generation [51.00761167842468]
  We propose a novel framework called RSL- that combines bidirectional schema linking, contextual information augmentation, binary selection strategy, and multi-turn self-correction. benchmarks demonstrate that our approach achieves SOTA execution accuracy among open-source solutions, with 67.2% on BIRD and 87.9% on GPT-4ocorrection. Our approach outperforms a series of GPT-4 based Text-to-Seek systems when adopting DeepSeek (much cheaper) with same intact prompts.
  arXiv  Detail & Related papers  (2024-10-31T16:22:26Z)
• Bridging Textual and Tabular Data for Cross-Domain Text-to-SQL Semantic Parsing [110.97778888305506]
  BRIDGE represents the question and DB schema in a tagged sequence where a subset of the fields are augmented with cell values mentioned in the question. BRIDGE attained state-of-the-art performance on popular cross-DB text-to- relational benchmarks. Our analysis shows that BRIDGE effectively captures the desired cross-modal dependencies and has the potential to generalize to more text-DB related tasks.
  arXiv  Detail & Related papers  (2020-12-23T12:33:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.