Is Large Language Model Good at Triple Set Prediction? An Empirical Study

• URL: http://arxiv.org/abs/2412.18443v1
• Date: Tue, 24 Dec 2024 14:03:07 GMT
• Title: Is Large Language Model Good at Triple Set Prediction? An Empirical Study
• Authors: Yuan Yuan, Yajing Xu, Wen Zhang,
• Abstract summary: The framework consists of LLM-based rule mining and LLM-based triple set prediction. The experimental results indicate that when LLMs are required to adhere to a large amount of factual knowledge to predict missing triples, significant hallucinations occurs, leading to a noticeable decline in performance.
• Score: 12.094218772036596
• License:
• Abstract: The core of the Knowledge Graph Completion (KGC) task is to predict and complete the missing relations or nodes in a KG. Common KGC tasks are mostly about inferring unknown elements with one or two elements being known in a triple. In comparison, the Triple Set Prediction (TSP) task is a more realistic knowledge graph completion task. It aims to predict all elements of unknown triples based on the information from known triples. In recent years, large language models (LLMs) have exhibited significant advancements in language comprehension, demonstrating considerable potential for KGC tasks. However, the potential of LLM on the TSP task has not yet to be investigated. Thus in this paper we proposed a new framework to explore the strengths and limitations of LLM in the TSP task. Specifically, the framework consists of LLM-based rule mining and LLM-based triple set prediction. The relation list of KG embedded within rich semantic information is first leveraged to prompt LLM in the generation of rules. This process is both efficient and independent of statistical information, making it easier to mine effective and realistic rules. For each subgraph, the specified rule is applied in conjunction with the relevant triples within that subgraph to guide the LLM in predicting the missing triples. Subsequently, the predictions from all subgraphs are consolidated to derive the complete set of predicted triples on KG. Finally, the method is evaluated on the relatively complete CFamily dataset. The experimental results indicate that when LLMs are required to adhere to a large amount of factual knowledge to predict missing triples, significant hallucinations occurs, leading to a noticeable decline in performance. To further explore the causes of this phenomenon, this paper presents a comprehensive analysis supported by a detailed case study.

Related papers

• KG-CF: Knowledge Graph Completion with Context Filtering under the Guidance of Large Language Models [55.39134076436266]
  KG-CF is a framework tailored for ranking-based knowledge graph completion tasks. KG-CF leverages LLMs' reasoning abilities to filter out irrelevant contexts, achieving superior results on real-world datasets.
  arXiv  Detail & Related papers  (2025-01-06T01:52:15Z)
• Can LLMs be Good Graph Judger for Knowledge Graph Construction? [33.958327252291]
  In this paper, we propose GraphJudger, a knowledge graph construction framework to address the aforementioned challenges. We introduce three innovative modules in our method, which are entity-centric iterative text denoising, knowledge aware instruction tuning and graph judgement. Experiments conducted on two general text-graph pair datasets and one domain-specific text-graph pair dataset show superior performances compared to baseline methods.
  arXiv  Detail & Related papers  (2024-11-26T12:46:57Z)
• GS-KGC: A Generative Subgraph-based Framework for Knowledge Graph Completion with Large Language Models [7.995716933782121]
  We propose a novel completion framework called textbfGenerative textbfSubgraph-based KGC (GS-KGC) This framework primarily includes a subgraph partitioning algorithm designed to generate negatives and neighbors. Experiments conducted on four common KGC datasets highlight the advantages of the proposed GS-KGC.
  arXiv  Detail & Related papers  (2024-08-20T13:13:41Z)
• Revisiting the Graph Reasoning Ability of Large Language Models: Case Studies in Translation, Connectivity and Shortest Path [53.71787069694794]
  We focus on the graph reasoning ability of Large Language Models (LLMs) We revisit the ability of LLMs on three fundamental graph tasks: graph description translation, graph connectivity, and the shortest-path problem. Our findings suggest that LLMs can fail to understand graph structures through text descriptions and exhibit varying performance for all these fundamental tasks.
  arXiv  Detail & Related papers  (2024-08-18T16:26:39Z)
• Start from Zero: Triple Set Prediction for Automatic Knowledge Graph Completion [49.19814695500355]
  We propose a graph-level automatic KG completion task called Triple Set Prediction (TSP) TSP assumes none of the elements in the missing triples is given. To tackle the huge candidate triples for prediction, we propose a novel and efficient subgraph-based method GPHT.
  arXiv  Detail & Related papers  (2024-06-26T08:26:32Z)
• Few-shot Knowledge Graph Relational Reasoning via Subgraph Adaptation [51.47994645529258]
  Few-shot Knowledge Graph (KG) Reasoning aims to predict unseen triplets (i.e., query triplets) for rare relations in KGs. We propose SAFER (Subgraph Adaptation for Few-shot Reasoning), a novel approach that effectively adapts the information in contextualized graphs to various subgraphs.
  arXiv  Detail & Related papers  (2024-06-19T21:40:35Z)
• Prompting Large Language Models with Knowledge Graphs for Question Answering Involving Long-tail Facts [50.06633829833144]
  Large Language Models (LLMs) are effective in performing various NLP tasks, but struggle to handle tasks that require extensive, real-world knowledge. We propose a benchmark that requires knowledge of long-tail facts for answering the involved questions. Our experiments show that LLMs alone struggle with answering these questions, especially when the long-tail level is high or rich knowledge is required.
  arXiv  Detail & Related papers  (2024-05-10T15:10:20Z)
• Chain-of-History Reasoning for Temporal Knowledge Graph Forecasting [32.711428457485596]
  Temporal Knowledge Graph (TKG) forecasting aims to predict future facts based on given histories. Most recent graph-based models excel at capturing structural information within TKGs but lack semantic comprehension abilities. We propose Chain-of-History (CoH) reasoning which explores high-order histories step-by-step.
  arXiv  Detail & Related papers  (2024-02-22T08:51:39Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.