EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution

• URL: http://arxiv.org/abs/2603.10697v1
• Date: Wed, 11 Mar 2026 12:10:03 GMT
• Title: EvoSchema: Towards Text-to-SQL Robustness Against Schema Evolution
• Abstract summary: We present Evo, a comprehensive benchmark designed to assess and enhance robustness of text-to- paraphrasing systems under real-world changes.<n>Through Evo, we conduct an in-depth evaluation spanning different open source and closed-source LLMs, revealing that table-level perturbations have a significantly greater impact on model performance compared to column-level changes.
• Score: 15.06978610447021
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Neural text-to-SQL models, which translate natural language questions (NLQs) into SQL queries given a database schema, have achieved remarkable performance. However, database schemas frequently evolve to meet new requirements. Such schema evolution often leads to performance degradation for models trained on static schemas. Existing work either mainly focuses on simply paraphrasing some syntactic or semantic mappings among NLQ, DB and SQL, or lacks a comprehensive and controllable way to investigate the model robustness issue under the schema evolution, which is insufficient when facing the increasingly complex and rich database schema changes in reality, especially in the LLM era. To address the challenges posed by schema evolution, we present EvoSchema, a comprehensive benchmark designed to assess and enhance the robustness of text-to-SQL systems under real-world schema changes. EvoSchema introduces a novel schema evolution taxonomy, encompassing ten perturbation types across columnlevel and table-level modifications, systematically simulating the dynamic nature of database schemas. Through EvoSchema, we conduct an in-depth evaluation spanning different open source and closed-source LLMs, revealing that table-level perturbations have a significantly greater impact on model performance compared to column-level changes. Furthermore, EvoSchema inspires the development of more resilient text-to-SQL systems, in terms of both model training and database design. The models trained on EvoSchema's diverse schema designs can force the model to distinguish the schema difference for the same questions to avoid learning spurious patterns, which demonstrate remarkable robustness compared to those trained on unperturbed data on average. This benchmark offers valuable insights into model behavior and a path forward for designing systems capable of thriving in dynamic, real-world environments.

Related papers

• The Power of Order: Fooling LLMs with Adversarial Table Permutations [49.84535222399142]
  Adversarial Table Permutation is a gradient-based attack that efficiently identifies worst-case permutations designed to maximally disrupt model performance.<n>Our experiments demonstrate that ATP significantly degrades the performance of a wide range of LLMs.<n>Our findings expose a fundamental weakness in how current LLMs process structured data, underscoring the urgent need to develop permutation-robust models for reliable, real-world applications.
  arXiv  Detail & Related papers  (2026-05-01T06:25:55Z)
• ASTRA: Adaptive Semantic Tree Reasoning Architecture for Complex Table Question Answering [55.55968342644846]
  Table serialization remains a critical bottleneck for Large Language Models (LLMs) in complex table question answering.<n>Existing serialization methods fail to capture explicit hierarchies and lack schema flexibility.<n>We propose ASTRA (Adaptive Semantic Tree Reasoning Architecture) including two main modules, AdaSTR and DuTR.<n> Experiments on complex table benchmarks demonstrate that our method achieves state-of-the-art (SOTA) performance.
  arXiv  Detail & Related papers  (2026-04-10T06:09:41Z)
• PluRel: Synthetic Data unlocks Scaling Laws for Relational Foundation Models [51.42043158297229]
  We introduce Pluel, a framework to synthesize multi-tabular relational databases from scratch.<n>In a step-by-step fashion, Pluel models (1) schemas with directed graphs, (2) inter-table primary-foreign key connectivity with bipartite graphs, and, (3) feature distributions in tables via conditional causal mechanisms.
  arXiv  Detail & Related papers  (2026-02-03T21:35:18Z)
• EvolSQL: Structure-Aware Evolution for Scalable Text-to-SQL Data Synthesis [25.689983072200047]
  Evol is a structure-aware data synthesis framework that evolves queries into richer and more semantically diverse forms.<n>A 7B model outperforms one trained on the much larger Syn dataset using only 1/18 of the data.
  arXiv  Detail & Related papers  (2026-01-08T12:19:50Z)
• Exploring Database Normalization Effects on SQL Generation [0.9256004093538199]
  normalization is a critical yet often overlooked factor in natural language tosql (NL2) systems.<n>We evaluate eight leading large language models on synthetic and real-world datasets with varied normalization levels.<n>Our findings suggest that the optimal schema design for NL2 applications depends on the types of queries to be supported.
  arXiv  Detail & Related papers  (2025-10-02T13:11:30Z)
• C2-Evo: Co-Evolving Multimodal Data and Model for Self-Improving Reasoning [78.36259648527401]
  C2-Evo is an automatic, closed-loop self-improving framework that jointly evolves both training data and model capabilities.<n>We show that C2-Evo consistently obtains considerable performance gains across multiple mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-07-22T12:27:08Z)
• LLM-Symbolic Integration for Robust Temporal Tabular Reasoning [69.27153114778748]
  We introduce TempTabQA-C, a synthetic dataset designed for systematic and controlled evaluations.<n>This structured approach allows Large Language Models (LLMs) to generate and executesql queries, enhancing generalization and mitigating biases.
  arXiv  Detail & Related papers  (2025-06-06T05:14:04Z)
• SchemaGraphSQL: Efficient Schema Linking with Pathfinding Graph Algorithms for Text-to-SQL on Large-Scale Databases [1.6544167074080365]
  We present a zero-shot, training-free schema linking approach that first constructs a schema graph based on foreign key relations.<n>We apply classical path-finding algorithms and post-processing to identify the optimal sequence of tables and columns that should be joined.<n>Our method achieves state-of-the-art results on the BIRD benchmark, outperforming previous specialized, fine-tuned, and complex multi-step LLM-based approaches.
  arXiv  Detail & Related papers  (2025-05-23T20:42:36Z)
• Text2Schema: Filling the Gap in Designing Database Table Structures based on Natural Language [22.15408079332362]
  People without a database background usually rely on file systems or tools such as Excel data management.<n> Database systems possess strong management capabilities, but require a high level of professional expertise from users.
  arXiv  Detail & Related papers  (2025-03-31T09:39:19Z)
• Matchmaker: Self-Improving Large Language Model Programs for Schema Matching [60.23571456538149]
  We propose a compositional language model program for schema matching, comprised of candidate generation, refinement and confidence scoring. Matchmaker self-improves in a zero-shot manner without the need for labeled demonstrations. Empirically, we demonstrate on real-world medical schema matching benchmarks that Matchmaker outperforms previous ML-based approaches.
  arXiv  Detail & Related papers  (2024-10-31T16:34:03Z)
• TableRAG: Million-Token Table Understanding with Language Models [53.039560091592215]
  TableRAG is a Retrieval-Augmented Generation (RAG) framework specifically designed for LM-based table understanding.<n>TableRAG leverages query expansion combined with schema and cell retrieval to pinpoint crucial information before providing it to the LMs.<n>Our results demonstrate that TableRAG achieves the highest retrieval quality, leading to the new state-of-the-art performance on large-scale table understanding.
  arXiv  Detail & Related papers  (2024-10-07T04:15:02Z)
• Compound Schema Registry [0.0]
  We propose the use of generalized schema evolution (GSE) facilitated by a compound AI system. This system employs Large Language Models (LLMs) to interpret the semantics of schema changes. Our approach includes developing a task-specific language, Transformation Language (STL), to generate schema mappings as an intermediate representation.
  arXiv  Detail & Related papers  (2024-06-17T05:50:46Z)
• MT-Teql: Evaluating and Augmenting Consistency of Text-to-SQL Models with Metamorphic Testing [11.566463879334862]
  We propose MT-Teql, a Metamorphic Testing-based framework for evaluating and augmenting the consistency of text-to-preserving models. Our framework exposes thousands of prediction errors from SOTA models and enriches existing datasets by order of magnitude, eliminating over 40% inconsistency errors without compromising standard accuracy.
  arXiv  Detail & Related papers  (2020-12-21T07:43:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.