Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL

• URL: http://arxiv.org/abs/2402.16567v3
• Date: Thu, 5 Sep 2024 06:34:11 GMT
• Title: Aligning Large Language Models to a Domain-specific Graph Database for NL2GQL
• Authors: Yuanyuan Liang, Keren Tan, Tingyu Xie, Wenbiao Tao, Siyuan Wang, Yunshi Lan, Weining Qian,
• Abstract summary: We present a well-defined pipeline for NL2GQL tasks tailored to a particular domain. We employ ChatGPT to generate NLGQL data pairs, leveraging the provided graph DB with self-instruction. We then employ the generated data to fine-tune LLMs, ensuring alignment between LLMs and the graph DB.
• Score: 16.637504932927616
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Graph Databases (Graph DB) find extensive application across diverse domains such as finance, social networks, and medicine. Yet, the translation of Natural Language (NL) into the Graph Query Language (GQL), referred to as NL2GQL, poses significant challenges owing to its intricate and specialized nature. Some approaches have sought to utilize Large Language Models (LLMs) to address analogous tasks like text2SQL. Nonetheless, in the realm of NL2GQL tasks tailored to a particular domain, the absence of domain-specific NL-GQL data pairs adds complexity to aligning LLMs with the graph DB. To tackle this challenge, we present a well-defined pipeline. Initially, we utilize ChatGPT to generate NL-GQL data pairs, leveraging the provided graph DB with self-instruction. Subsequently, we employ the generated data to fine-tune LLMs, ensuring alignment between LLMs and the graph DB. Moreover, we find the importance of relevant schema in efficiently generating accurate GQLs. Thus, we introduce a method to extract relevant schema as the input context. We evaluate our method using two carefully constructed datasets derived from graph DBs in the finance and medicine domains, named FinGQL and MediGQL. Experimental results reveal that our approach significantly outperforms a set of baseline methods, with improvements of 5.90 and 6.36 absolute points on EM, and 6.00 and 7.09 absolute points on EX for FinGQL and MediGQL, respectively.

Related papers

• CypherBench: Towards Precise Retrieval over Full-scale Modern Knowledge Graphs in the LLM Era [4.369550829556578]
  We introduce CypherBench, the first benchmark with 11 large-scale, multi-domain property graphs with 7.8 million entities and over 10,000 questions. We propose property graph views on top of the underlying RDF graph that can be efficiently queried by LLMs using Cypher.
  arXiv  Detail & Related papers  (2024-12-24T23:22:04Z)
• NAT-NL2GQL: A Novel Multi-Agent Framework for Translating Natural Language to Graph Query Language [13.661054027428868]
  We propose NAT-NL2GQL, a novel framework for translating natural language to graph query language. Our framework consists of three synergistic agents: the Preprocessor agent, the Generator agent, and the Refiner agent. Given the scarcity of high-quality open-source NL2GQL datasets based on nGQL syntax, we developed StockGQL, a dataset constructed from a financial market graph database.
  arXiv  Detail & Related papers  (2024-12-11T04:14:09Z)
• Towards Evaluating Large Language Models for Graph Query Generation [49.49881799107061]
  Large Language Models (LLMs) are revolutionizing the landscape of Generative Artificial Intelligence (GenAI) This paper presents a comparative study addressing the challenge of generating queries a powerful language for interacting with graph databases using open-access LLMs. Our empirical analysis of query generation accuracy reveals that Claude Sonnet 3.5 outperforms its counterparts in this specific domain.
  arXiv  Detail & Related papers  (2024-11-13T09:11:56Z)
• Less is More: Making Smaller Language Models Competent Subgraph Retrievers for Multi-hop KGQA [51.3033125256716]
  We model the subgraph retrieval task as a conditional generation task handled by small language models. Our base generative subgraph retrieval model, consisting of only 220M parameters, competitive retrieval performance compared to state-of-the-art models. Our largest 3B model, when plugged with an LLM reader, sets new SOTA end-to-end performance on both the WebQSP and CWQ benchmarks.
  arXiv  Detail & Related papers  (2024-10-08T15:22:36Z)
• Can Large Language Models Analyze Graphs like Professionals? A Benchmark, Datasets and Models [90.98855064914379]
  We introduce ProGraph, a benchmark for large language models (LLMs) to process graphs. Our findings reveal that the performance of current LLMs is unsatisfactory, with the best model achieving only 36% accuracy. We propose LLM4Graph datasets, which include crawled documents and auto-generated codes based on 6 widely used graph libraries.
  arXiv  Detail & Related papers  (2024-09-29T11:38:45Z)
• Graph Chain-of-Thought: Augmenting Large Language Models by Reasoning on Graphs [60.71360240206726]
  Large language models (LLMs) suffer from hallucinations, especially on knowledge-intensive tasks. Existing works propose to augment LLMs with individual text units retrieved from external knowledge corpora. We propose a framework called Graph Chain-of-thought (Graph-CoT) to augment LLMs with graphs by encouraging LLMs to reason on the graph iteratively.
  arXiv  Detail & Related papers  (2024-04-10T15:41:53Z)
• NL2KQL: From Natural Language to Kusto Query [1.7931930942711818]
  NL2KQL is an innovative framework that uses large language models (LLMs) to convert natural language queries (NLQs) to Kusto Query Language (KQL) queries. To validate NL2KQL's performance, we utilize an array of online (based on query execution) and offline (based on query parsing) metrics.
  arXiv  Detail & Related papers  (2024-04-03T01:09:41Z)
• SPARQL Generation: an analysis on fine-tuning OpenLLaMA for Question Answering over a Life Science Knowledge Graph [0.0]
  We evaluate strategies for fine-tuning the OpenLlama LLM for question answering over life science knowledge graphs. We propose an end-to-end data augmentation approach for extending a set of existing queries over a given knowledge graph. We also investigate the role of semantic "clues" in the queries, such as meaningful variable names and inline comments.
  arXiv  Detail & Related papers  (2024-02-07T07:24:01Z)
• $R^3$-NL2GQL: A Model Coordination and Knowledge Graph Alignment Approach for NL2GQL [45.13624736815995]
  We introduce a novel approach, $R3$-NL2GQL, integrating both small and large Foundation Models for ranking, rewriting, and refining tasks. We have developed a bilingual dataset, sourced from graph database manuals and selected open-source Knowledge Graphs (KGs)
  arXiv  Detail & Related papers  (2023-11-03T12:11:12Z)
• Integrating Graphs with Large Language Models: Methods and Prospects [68.37584693537555]
  Large language models (LLMs) have emerged as frontrunners, showcasing unparalleled prowess in diverse applications. Merging the capabilities of LLMs with graph-structured data has been a topic of keen interest. This paper bifurcates such integrations into two predominant categories.
  arXiv  Detail & Related papers  (2023-10-09T07:59:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.