LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities

• URL: http://arxiv.org/abs/2305.13168v4
• Date: Thu, 26 Dec 2024 18:54:53 GMT
• Title: LLMs for Knowledge Graph Construction and Reasoning: Recent Capabilities and Future Opportunities
• Authors: Yuqi Zhu, Xiaohan Wang, Jing Chen, Shuofei Qiao, Yixin Ou, Yunzhi Yao, Shumin Deng, Huajun Chen, Ningyu Zhang,
• Abstract summary: Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning evaluated. We propose AutoKG, a multi-agent-based approach employing LLMs and external sources for KG construction and reasoning.
• Score: 66.36633042421387
• License:
• Abstract: This paper presents an exhaustive quantitative and qualitative evaluation of Large Language Models (LLMs) for Knowledge Graph (KG) construction and reasoning. We engage in experiments across eight diverse datasets, focusing on four representative tasks encompassing entity and relation extraction, event extraction, link prediction, and question-answering, thereby thoroughly exploring LLMs' performance in the domain of construction and inference. Empirically, our findings suggest that LLMs, represented by GPT-4, are more suited as inference assistants rather than few-shot information extractors. Specifically, while GPT-4 exhibits good performance in tasks related to KG construction, it excels further in reasoning tasks, surpassing fine-tuned models in certain cases. Moreover, our investigation extends to the potential generalization ability of LLMs for information extraction, leading to the proposition of a Virtual Knowledge Extraction task and the development of the corresponding VINE dataset. Based on these empirical findings, we further propose AutoKG, a multi-agent-based approach employing LLMs and external sources for KG construction and reasoning. We anticipate that this research can provide invaluable insights for future undertakings in the field of knowledge graphs. The code and datasets are in https://github.com/zjunlp/AutoKG.

Related papers

• MAmmoTH-VL: Eliciting Multimodal Reasoning with Instruction Tuning at Scale [66.73529246309033]
  multimodal large language models (MLLMs) have shown significant potential in a broad range of multimodal tasks. Existing instruction-tuning datasets only provide phrase-level answers without any intermediate rationales. We introduce a scalable and cost-effective method to construct a large-scale multimodal instruction-tuning dataset with rich intermediate rationales.
  arXiv  Detail & Related papers  (2024-12-06T18:14:24Z)
• Aggregated Knowledge Model: Enhancing Domain-Specific QA with Fine-Tuned and Retrieval-Augmented Generation Models [0.0]
  This paper introduces a novel approach to enhancing closed-domain Question Answering (QA) systems. It focuses on the specific needs of the Lawrence Berkeley National Laboratory (LBL) Science Information Technology (ScienceIT) domain.
  arXiv  Detail & Related papers  (2024-10-24T00:49:46Z)
• MMSci: A Dataset for Graduate-Level Multi-Discipline Multimodal Scientific Understanding [59.41495657570397]
  We present a comprehensive dataset compiled from Nature Communications articles covering 72 scientific fields. We evaluated 19 proprietary and open-source models on two benchmark tasks, figure captioning and multiple-choice, and conducted human expert annotation. Fine-tuning Qwen2-VL-7B with our task-specific data achieved better performance than GPT-4o and even human experts in multiple-choice evaluations.
  arXiv  Detail & Related papers  (2024-07-06T00:40:53Z)
• Explore then Determine: A GNN-LLM Synergy Framework for Reasoning over Knowledge Graph [38.31983923708175]
  This paper focuses on the Question Answering over Knowledge Graph (KGQA) task. It proposes an Explore-then-Determine (EtD) framework that synergizes Large Language Models with graph neural networks (GNNs) for reasoning over KGs. EtD achieves state-of-the-art performance and generates faithful reasoning results.
  arXiv  Detail & Related papers  (2024-06-03T09:38:28Z)
• Exploring the Potential of Large Language Models in Graph Generation [51.046188600990014]
  Graph generation requires large language models (LLMs) to generate graphs with given properties. This paper explores the abilities of LLMs for graph generation with systematical task designs and experiments. Our evaluations demonstrate that LLMs, particularly GPT-4, exhibit preliminary abilities in graph generation tasks.
  arXiv  Detail & Related papers  (2024-03-21T12:37:54Z)
• Had enough of experts? Quantitative knowledge retrieval from large language models [4.091195951668217]
  Large language models (LLMs) have been extensively studied for their abilities to generate convincing natural language sequences. We introduce a framework that leverages LLMs to enhance Bayesian models by eliciting expert-like prior knowledge and imputing missing data.
  arXiv  Detail & Related papers  (2024-02-12T16:32:37Z)
• Mining experimental data from Materials Science literature with Large Language Models: an evaluation study [1.9849264945671101]
  This study is dedicated to assessing the capabilities of large language models (LLMs) in extracting structured information from scientific documents in materials science. We focus on two critical tasks of information extraction: (i) a named entity recognition (NER) of studied materials and physical properties and (ii) a relation extraction (RE) between these entities. The performance of LLMs in executing these tasks is benchmarked against traditional models based on the BERT architecture and rule-based approaches (baseline)
  arXiv  Detail & Related papers  (2024-01-19T23:00:31Z)
• Knowledge Plugins: Enhancing Large Language Models for Domain-Specific Recommendations [50.81844184210381]
  We propose a general paradigm that augments large language models with DOmain-specific KnowledgE to enhance their performance on practical applications, namely DOKE. This paradigm relies on a domain knowledge extractor, working in three steps: 1) preparing effective knowledge for the task; 2) selecting the knowledge for each specific sample; and 3) expressing the knowledge in an LLM-understandable way.
  arXiv  Detail & Related papers  (2023-11-16T07:09:38Z)
• Can LLMs Augment Low-Resource Reading Comprehension Datasets? Opportunities and Challenges [3.130575840003799]
  GPT-4 can be used to augment existing reading comprehension datasets. This work serves to be the first analysis of LLMs as synthetic data augmenters for QA systems.
  arXiv  Detail & Related papers  (2023-09-21T18:48:02Z)
• Knowledge-Augmented Reasoning Distillation for Small Language Models in Knowledge-Intensive Tasks [90.11273439036455]
  Large Language Models (LLMs) have shown promising performance in knowledge-intensive reasoning tasks. We propose Knowledge-Augmented Reasoning Distillation (KARD), a novel method that fine-tunes small LMs to generate rationales from LLMs with augmented knowledge retrieved from an external knowledge base. We empirically show that KARD significantly improves the performance of small T5 and GPT models on the challenging knowledge-intensive reasoning datasets.
  arXiv  Detail & Related papers  (2023-05-28T13:00:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.