DBCopilot: Natural Language Querying over Massive Databases via Schema Routing

• URL: http://arxiv.org/abs/2312.03463v3
• Date: Sat, 22 Feb 2025 08:32:05 GMT
• Title: DBCopilot: Natural Language Querying over Massive Databases via Schema Routing
• Authors: Tianshu Wang, Xiaoyang Chen, Hongyu Lin, Xianpei Han, Le Sun, Hao Wang, Zhenyu Zeng,
• Abstract summary: We present DBCopilot, a framework that addresses challenges by employing a compact and flexible copilot model for routing over massive databases.<n>This framework utilizes a single lightweight differentiable search index to construct semantic mappings for massive database schemata, and navigates natural language questions to their target databases and tables in a relation joint retrieval manner.
• Score: 47.009638761948466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The development of Natural Language Interfaces to Databases (NLIDBs) has been greatly advanced by the advent of large language models (LLMs), which provide an intuitive way to translate natural language (NL) questions into Structured Query Language (SQL) queries. While significant progress has been made in LLM-based NL2SQL, existing approaches face several challenges in real-world scenarios of natural language querying over massive databases. In this paper, we present DBCopilot, a framework that addresses these challenges by employing a compact and flexible copilot model for routing over massive databases. Specifically, DBCopilot decouples schema-agnostic NL2SQL into schema routing and SQL generation. This framework utilizes a single lightweight differentiable search index to construct semantic mappings for massive database schemata, and navigates natural language questions to their target databases and tables in a relation-aware joint retrieval manner. The routed schemata and questions are then fed into LLMs for effective SQL generation. Furthermore, DBCopilot introduces a reverse schema-to-question generation paradigm that can automatically learn and adapt the router over massive databases without manual intervention. Experimental results verify that DBCopilot is a scalable and effective solution for schema-agnostic NL2SQL, providing a significant advance in handling natural language querying over massive databases for NLIDBs.

Related papers

• DB-Explore: Automated Database Exploration and Instruction Synthesis for Text-to-SQL [18.915121803834698]
  We propose DB-Explore, a novel framework for database understanding using large language models (LLMs) Our framework enables comprehensive database understanding through diverse sampling strategies and automated instruction generation. Our open-source implementation, based on Qwen2.5-coder-7B model, outperforms multiple GPT-4-driven text-to-coder systems in comparative evaluations.
  arXiv  Detail & Related papers  (2025-03-06T20:46:43Z)
• E-SQL: Direct Schema Linking via Question Enrichment in Text-to-SQL [1.187832944550453]
  We introduce E- repository, a novel pipeline designed to address challenges through direct schema linking and candidate predicate augmentation. E- enhances the natural language query by incorporating relevant database items (i.e. tables, columns, and values) and conditions directly into the question, bridging the gap between the query and the database structure. We investigate the impact of schema filtering, a technique widely explored in previous work, and demonstrate its diminishing returns when applied alongside advanced large language models.
  arXiv  Detail & Related papers  (2024-09-25T09:02:48Z)
• Text2SQL is Not Enough: Unifying AI and Databases with TAG [47.45480855418987]
  Table-Augmented Generation (TAG) is a paradigm for answering natural language questions over databases. We develop benchmarks to study the TAG problem and find that standard methods answer no more than 20% of queries correctly.
  arXiv  Detail & Related papers  (2024-08-27T00:50:14Z)
• Relational Database Augmented Large Language Model [59.38841050766026]
  Large language models (LLMs) excel in many natural language processing (NLP) tasks. They can only incorporate new knowledge through training or supervised fine-tuning processes. This precise, up-to-date, and private information is typically stored in relational databases.
  arXiv  Detail & Related papers  (2024-07-21T06:19:10Z)
• RB-SQL: A Retrieval-based LLM Framework for Text-to-SQL [48.516004807486745]
  Large language models (LLMs) with in-context learning have significantly improved the performance of text-to- task. We propose RB-, a novel retrieval-based framework for in-context prompt engineering. Experiment results demonstrate that our model achieves better performance than several competitive baselines on public datasets BIRD and Spider.
  arXiv  Detail & Related papers  (2024-07-11T08:19:58Z)
• Next-Generation Database Interfaces: A Survey of LLM-based Text-to-SQL [15.75829309721909]
  Generating accuratesql from natural language questions (text-to-) is a long-standing challenge. PLMs have been developed and utilized for text-to- tasks, achieving promising performance. Recently, large language models (LLMs) have demonstrated significant capabilities in natural language understanding.
  arXiv  Detail & Related papers  (2024-06-12T17:13:17Z)
• Semantic Parsing for Complex Data Retrieval: Targeting Query Plans vs. SQL for No-Code Access to Relational Databases [2.933060994339853]
  We investigate the potential of an alternative query language with simpler syntax and modular specification of complex queries. The proposed alternative query language is called Query Plan Language (QPL) We present ways to address the challenge of complex queries in an iterative, user-controlled manner.
  arXiv  Detail & Related papers  (2023-12-22T16:16:15Z)
• SQL-PaLM: Improved Large Language Model Adaptation for Text-to-SQL (extended) [53.95151604061761]
  This paper introduces the framework for enhancing Text-to- filtering using large language models (LLMs) With few-shot prompting, we explore the effectiveness of consistency decoding with execution-based error analyses. With instruction fine-tuning, we delve deep in understanding the critical paradigms that influence the performance of tuned LLMs.
  arXiv  Detail & Related papers  (2023-05-26T21:39:05Z)
• Prompting GPT-3.5 for Text-to-SQL with De-semanticization and Skeleton Retrieval [17.747079214502673]
  Text-to- is a task that converts a natural language question into a structured query language () to retrieve information from a database. In this paper, we propose an LLM-based framework for Text-to- which retrieves helpful demonstration examples to prompt LLMs. We design a de-semanticization mechanism that extracts question skeletons, allowing us to retrieve similar examples based on their structural similarity.
  arXiv  Detail & Related papers  (2023-04-26T06:02:01Z)
• xDBTagger: Explainable Natural Language Interface to Databases Using Keyword Mappings and Schema Graph [0.17188280334580192]
  Translating natural language queries into structured query language (NLQ) in interfaces to relational databases is a challenging task. We propose xDBTagger, an explainable hybrid translation pipeline that explains the decisions made along the way to the user both textually and visually. xDBTagger is effective in terms of accuracy and translates the queries more efficiently compared to other state-of-the-art pipeline-based systems up to 10000 times.
  arXiv  Detail & Related papers  (2022-10-07T18:17:09Z)
• A Survey on Text-to-SQL Parsing: Concepts, Methods, and Future Directions [102.8606542189429]
  The goal of text-to-corpora parsing is to convert a natural language (NL) question to its corresponding structured query language () based on the evidences provided by databases. Deep neural networks have significantly advanced this task by neural generation models, which automatically learn a mapping function from an input NL question to an output query.
  arXiv  Detail & Related papers  (2022-08-29T14:24:13Z)
• Proton: Probing Schema Linking Information from Pre-trained Language Models for Text-to-SQL Parsing [66.55478402233399]
  We propose a framework to elicit relational structures via a probing procedure based on Poincar'e distance metric. Compared with commonly-used rule-based methods for schema linking, we found that probing relations can robustly capture semantic correspondences. Our framework sets new state-of-the-art performance on three benchmarks.
  arXiv  Detail & Related papers  (2022-06-28T14:05:25Z)
• Translating synthetic natural language to database queries: a polyglot deep learning framework [0.0]
  Polyglotter supports the mapping of natural language searches to database queries. It does not require the creation of manually annotated data for training. Our results indicate that our framework performs well on both synthetic and real databases.
  arXiv  Detail & Related papers  (2021-04-14T17:43:51Z)
• "What Do You Mean by That?" A Parser-Independent Interactive Approach for Enhancing Text-to-SQL [49.85635994436742]
  We include human in the loop and present a novel-independent interactive approach (PIIA) that interacts with users using multi-choice questions. PIIA is capable of enhancing the text-to-domain performance with limited interaction turns by using both simulation and human evaluation.
  arXiv  Detail & Related papers  (2020-11-09T02:14:33Z)
• Photon: A Robust Cross-Domain Text-to-SQL System [189.1405317853752]
  We present Photon, a robust, modular, cross-domain NLIDB that can flag natural language input to which a mapping cannot be immediately determined. The proposed method effectively improves the robustness of text-to-native system against untranslatable user input.
  arXiv  Detail & Related papers  (2020-07-30T07:44:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.