Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents

• URL: http://arxiv.org/abs/2509.24405v1
• Date: Mon, 29 Sep 2025 07:50:39 GMT
• Title: Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents
• Authors: Khanh Trinh Pham, Thu Huong Nguyen, Jun Jo, Quoc Viet Hung Nguyen, Thanh Tam Nguyen,
• Abstract summary: Text-to-Spider enables natural access to databases, yet most benchmarks are English-only, limiting multilingual progress.<n>We introduce MultiSpider 2.0, extending Spider 2.0 to eight languages (English, German, French, Spanish, Portuguese, Japanese, Chinese, Vietnamese)<n>On this benchmark, state-of-the-art LLMs reach only 4% execution accuracy when relying on intrinsic reasoning, versus 60% on MultiSpider 1.0.
• Score: 12.06248425636
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Text-to-SQL enables natural access to databases, yet most benchmarks are English-only, limiting multilingual progress. We introduce MultiSpider 2.0, extending Spider 2.0 to eight languages (English, German, French, Spanish, Portuguese, Japanese, Chinese, Vietnamese). It preserves Spider 2.0's structural difficulty while adding linguistic and dialectal variability, demanding deeper reasoning for complex SQL. On this benchmark, state-of-the-art LLMs (such as DeepSeek-R1 and OpenAI o1) reach only 4\% execution accuracy when relying on intrinsic reasoning, versus 60\% on MultiSpider 1.0. Therefore, we provide a collaboration-driven language agents baseline that iteratively refines queries, improving accuracy to 15\%. These results reveal a substantial multilingual gap and motivate methods that are robust across languages and ready for real-world enterprise deployment. Our benchmark is available at https://github.com/phkhanhtrinh23/Multilingual_Text_to_SQL.

Related papers

• A Multi-Language Object-Oriented Programming Benchmark for Large Language Models [61.267115598083315]
  A survey of 35 existing benchmarks uncovers three major imbalances.<n>85.7% focus on a single programming language.<n>94.3% target only function-level or statement-level tasks.<n>Over 80% include fewer than ten test cases on average.
  arXiv  Detail & Related papers  (2025-09-30T11:30:08Z)
• Text2Cypher Across Languages: Evaluating and Finetuning LLMs [0.17188280334580197]
  This paper investigates the performance of both foundational and finetuned LLMs on the Text2Cypher task across multiple languages.<n>We create and release a multilingual dataset by translating English questions into Spanish and Turkish.<n>Using standardized prompts and metrics, we evaluate several foundational models and observe a consistent performance pattern.
  arXiv  Detail & Related papers  (2025-06-26T16:31:10Z)
• MultiTEND: A Multilingual Benchmark for Natural Language to NoSQL Query Translation [6.142748564599452]
  This paper introduces MultiTEND, the first largest multilingual benchmark for natural language to query generation.<n>We analyze challenges in translating natural language to queries across diverse linguistic structures.<n>We introduce MultiLink, a novel framework that bridges the multilingual input to query generation gap through a Parallel Linking Process.
  arXiv  Detail & Related papers  (2025-02-16T07:12:47Z)
• Spider 2.0: Evaluating Language Models on Real-World Enterprise Text-to-SQL Workflows [64.94146689665628]
  Spider 2.0 is an evaluation framework for real-world text-to-sql problems derived from enterprise-level database use cases.<n>The databases in Spider 2.0 are sourced from real data applications, often containing over 1,000 columns and stored in local or cloud database systems such as BigQuery and Snowflake.<n>We show that solving problems in Spider 2.0 frequently requires understanding and searching through database metadata, dialect documentation, and even project-levels.
  arXiv  Detail & Related papers  (2024-11-12T12:52:17Z)
• CodeS: Towards Building Open-source Language Models for Text-to-SQL [42.11113113574589]
  We introduce CodeS, a series of pre-trained language models with parameters ranging from 1B to 15B. CodeS is a fully open language model, which achieves superior accuracy with much smaller parameter sizes. We conduct comprehensive evaluations on multiple datasets, including the widely used Spider benchmark.
  arXiv  Detail & Related papers  (2024-02-26T07:00:58Z)
• Ar-Spider: Text-to-SQL in Arabic [11.463438573648297]
  This paper introduces Ar-Spider 1, the first Arabic cross-language text-to-domain dataset. Due to the unique nature of the language, two major challenges have been encountered, namely linguistic and structural challenges. We propose the similarity relationship (CSR) approach, which results in a significant increase in the overall performance of about 1.52% for S2 and 1.06% for LGE and closes the gap between Arabic and English languages to 7.73%.
  arXiv  Detail & Related papers  (2024-02-22T23:11:17Z)
• Breaking Language Barriers in Multilingual Mathematical Reasoning: Insights and Observations [59.056367787688146]
  This paper pioneers exploring and training powerful Multilingual Math Reasoning (xMR) LLMs. We construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages. By utilizing translation, we construct the first multilingual math reasoning instruction dataset, MGSM8KInstruct, encompassing ten distinct languages.
  arXiv  Detail & Related papers  (2023-10-31T08:09:20Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing [48.216386761482525]
  We present MultiSpider, the largest multilingual text-to- schema- dataset which covers seven languages (English, German, French, Spanish, Japanese, Chinese, and Vietnamese) Experimental results under three typical settings (zero-shot, monolingual and multilingual) reveal a 6.1% absolute drop in accuracy in non-English languages. We also propose a simple framework augmentation framework SAVe (Augmentation-with-Verification) which boosts the overall performance by about 1.8% and closes the 29.5% performance gap across languages.
  arXiv  Detail & Related papers  (2022-12-27T13:58:30Z)
• XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing [70.40401197026925]
  In-context learning using large language models has recently shown surprising results for semantic parsing tasks. This work introduces the XRICL framework, which learns to retrieve relevant English exemplars for a given query. We also include global translation exemplars for a target language to facilitate the translation process for large language models.
  arXiv  Detail & Related papers  (2022-10-25T01:33:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.