IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages

• URL: http://arxiv.org/abs/2604.13686v1
• Date: Wed, 15 Apr 2026 10:07:37 GMT
• Title: IndicDB -- Benchmarking Multilingual Text-to-SQL Capabilities in Indian Languages
• Abstract summary: IndicDB is a benchmark for evaluating cross-lingual semantic parsing across diverse Indic languages.<n> relational schemas are sourced from open-data platforms, includingNDAP and the India Data Portal.<n>IndicDB comprises 20 databases across 237 tables, generating 15,617 tasks across English, Hindi, and five Indic languages.
• Score: 5.678442494739663
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While Large Language Models (LLMs) have significantly advanced Text-to-SQL performance, existing benchmarks predominantly focus on Western contexts and simplified schemas, leaving a gap in real-world, non-Western applications. We present IndicDB, a multilingual Text-to-SQL benchmark for evaluating cross-lingual semantic parsing across diverse Indic languages. The relational schemas are sourced from open-data platforms, including the National Data and Analytics Platform (NDAP) and the India Data Portal (IDP), ensuring realistic administrative data complexity. IndicDB comprises 20 databases across 237 tables. To convert denormalized government data into rich relational structures, we employ an iterative three-agent framework (Architect, Auditor, Refiner) to ensure structural rigor and high relational density (11.85 tables per database; join depths up to six). Our pipeline is value-aware, difficulty-calibrated, and join-enforced, generating 15,617 tasks across English, Hindi, and five Indic languages. We evaluate cross-lingual semantic parsing performance of state-of-the-art models (DeepSeek v3.2, MiniMax 2.7, LLaMA 3.3, Qwen3) across seven linguistic variants. Results show a 9.00% performance drop from English to Indic languages, revealing an "Indic Gap" driven by harder schema linking, increased structural ambiguity, and limited external knowledge. IndicDB serves as a rigorous benchmark for multilingual Text-to-SQL. Code and data: https://anonymous.4open.science/r/multilingualText2Sql-Indic--DDCC/

Related papers

• ProSPy: A Profiling-Driven SQL-Python Agentic Framework for Enterprise Text-to-SQL [51.98164069124653]
  ProSPy is a Profiling-driven framework for enterprise-scale Text-to-gnostic analysis.<n>ProSPy structures the reasoning process into four stages.<n>It first extracts fine-grained data evidence through automatic profiling.<n>It prunes large schemas into task-relevant contexts.
  arXiv  Detail & Related papers  (2026-06-04T08:13:05Z)
• EntSQL: A Benchmark for Grounding Text-to-SQL in Long-Context Enterprise Knowledge [20.59995531311023]
  Existing benchmarks such as Spider, BIRD, and Spider2.0 evaluate schema generalization, large-scale databases, and realistic.<n>We introduce Ent, an enterprise-oriented Text-to-private benchmark for evaluating long-context grounding over proprietary business documents.<n>On English inputs, the best evaluated system reaches only 15.9% when long-form documents are provided.
  arXiv  Detail & Related papers  (2026-06-02T09:12:57Z)
• Track-SQL: Enhancing Generative Language Models with Dual-Extractive Modules for Schema and Context Tracking in Multi-turn Text-to-SQL [50.84251174559123]
  Generative language models' performance does not extend equivalently to multi-turn Text-to-sourced.<n>We propose Track-, which enhances generative language models with dual-extractive modules designed to track schema and contextual changes in multi-turn Text-to-sourced.<n>Track- significantly improves execution accuracy in multi-turn interactions by 7.1% and 9.55% on datasets, respectively.
  arXiv  Detail & Related papers  (2026-03-06T07:57:06Z)
• Multilingual Text-to-SQL: Benchmarking the Limits of Language Models with Collaborative Language Agents [12.06248425636]
  Text-to-Spider enables natural access to databases, yet most benchmarks are English-only, limiting multilingual progress.<n>We introduce MultiSpider 2.0, extending Spider 2.0 to eight languages (English, German, French, Spanish, Portuguese, Japanese, Chinese, Vietnamese)<n>On this benchmark, state-of-the-art LLMs reach only 4% execution accuracy when relying on intrinsic reasoning, versus 60% on MultiSpider 1.0.
  arXiv  Detail & Related papers  (2025-09-29T07:50:39Z)
• MultiTEND: A Multilingual Benchmark for Natural Language to NoSQL Query Translation [6.142748564599452]
  This paper introduces MultiTEND, the first largest multilingual benchmark for natural language to query generation.<n>We analyze challenges in translating natural language to queries across diverse linguistic structures.<n>We introduce MultiLink, a novel framework that bridges the multilingual input to query generation gap through a Parallel Linking Process.
  arXiv  Detail & Related papers  (2025-02-16T07:12:47Z)
• UNITE: A Unified Benchmark for Text-to-SQL Evaluation [72.72040379293718]
  We introduce a UNIfied benchmark for Text-to-domain systems. It is composed of publicly available text-to-domain datasets and 29K databases. Compared to the widely used Spider benchmark, we introduce a threefold increase in SQL patterns.
  arXiv  Detail & Related papers  (2023-05-25T17:19:52Z)
• Can LLM Already Serve as A Database Interface? A BIg Bench for Large-Scale Database Grounded Text-to-SQLs [89.68522473384522]
  We present Bird, a big benchmark for large-scale database grounded in text-to-efficient tasks. Our emphasis on database values highlights the new challenges of dirty database contents. Even the most effective text-to-efficient models, i.e. ChatGPT, achieves only 40.08% in execution accuracy.
  arXiv  Detail & Related papers  (2023-05-04T19:02:29Z)
• MultiSpider: Towards Benchmarking Multilingual Text-to-SQL Semantic Parsing [48.216386761482525]
  We present MultiSpider, the largest multilingual text-to- schema- dataset which covers seven languages (English, German, French, Spanish, Japanese, Chinese, and Vietnamese) Experimental results under three typical settings (zero-shot, monolingual and multilingual) reveal a 6.1% absolute drop in accuracy in non-English languages. We also propose a simple framework augmentation framework SAVe (Augmentation-with-Verification) which boosts the overall performance by about 1.8% and closes the 29.5% performance gap across languages.
  arXiv  Detail & Related papers  (2022-12-27T13:58:30Z)
• XRICL: Cross-lingual Retrieval-Augmented In-Context Learning for Cross-lingual Text-to-SQL Semantic Parsing [70.40401197026925]
  In-context learning using large language models has recently shown surprising results for semantic parsing tasks. This work introduces the XRICL framework, which learns to retrieve relevant English exemplars for a given query. We also include global translation exemplars for a target language to facilitate the translation process for large language models.
  arXiv  Detail & Related papers  (2022-10-25T01:33:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.