EntGPT: Linking Generative Large Language Models with Knowledge Bases

• URL: http://arxiv.org/abs/2402.06738v1
• Date: Fri, 9 Feb 2024 19:16:27 GMT
• Title: EntGPT: Linking Generative Large Language Models with Knowledge Bases
• Authors: Yifan Ding, Amrit Poudel, Qingkai Zeng, Tim Weninger, Balaji Veeramani, Sanmitra Bhattacharya
• Abstract summary: The ability of Large Language Models to generate factually correct output remains relatively unexplored. We design a three-step hard-prompting method to probe LLMs' ED performance without supervised fine-tuning. We further improve the knowledge grounding ability through instruction tuning (IT) with similar prompts and responses.
• Score: 9.067856411512427
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The ability of Large Language Models (LLMs) to generate factually correct output remains relatively unexplored due to the lack of fact-checking and knowledge grounding during training and inference. In this work, we aim to address this challenge through the Entity Disambiguation (ED) task. We first consider prompt engineering, and design a three-step hard-prompting method to probe LLMs' ED performance without supervised fine-tuning (SFT). Overall, the prompting method improves the micro-F_1 score of the original vanilla models by a large margin, on some cases up to 36% and higher, and obtains comparable performance across 10 datasets when compared to existing methods with SFT. We further improve the knowledge grounding ability through instruction tuning (IT) with similar prompts and responses. The instruction-tuned model not only achieves higher micro-F1 score performance as compared to several baseline methods on supervised entity disambiguation tasks with an average micro-F_1 improvement of 2.1% over the existing baseline models, but also obtains higher accuracy on six Question Answering (QA) tasks in the zero-shot setting. Our methodologies apply to both open- and closed-source LLMs.

Related papers

• CommonIT: Commonality-Aware Instruction Tuning for Large Language Models via Data Partitions [17.252582058787937]
  We introduce a novel instruction tuning strategy termed CommonIT: Commonality-aware Instruction Tuning. Specifically, we cluster instruction datasets into distinct groups with three proposed metrics (Task, Embedding and Length) Rigorous testing on LLaMa models demonstrates CommonIT's effectiveness in enhancing the instruction-following capabilities of LLMs.
  arXiv  Detail & Related papers  (2024-10-04T01:42:35Z)
• ChatEL: Entity Linking with Chatbots [11.944348800783834]
  ChatEL is a three-step framework to prompt Large Language Models to return accurate results. Overall the ChatEL framework improves the average F1 performance across 10 datasets by more than 2%.
  arXiv  Detail & Related papers  (2024-02-20T20:52:57Z)
• kNN-ICL: Compositional Task-Oriented Parsing Generalization with Nearest Neighbor In-Context Learning [50.40636157214161]
  Task-Oriented Parsing (TOP) enables conversational assistants to interpret user commands expressed in natural language. LLMs have achieved impressive performance in computer programs based on a natural language prompt. This paper focuses on harnessing the capabilities of LLMs for semantic parsing tasks.
  arXiv  Detail & Related papers  (2023-12-17T17:26:50Z)
• BOOST: Harnessing Black-Box Control to Boost Commonsense in LMs' Generation [60.77990074569754]
  We present a computation-efficient framework that steers a frozen Pre-Trained Language Model towards more commonsensical generation. Specifically, we first construct a reference-free evaluator that assigns a sentence with a commonsensical score. We then use the scorer as the oracle for commonsense knowledge, and extend the controllable generation method called NADO to train an auxiliary head.
  arXiv  Detail & Related papers  (2023-10-25T23:32:12Z)
• Generate then Select: Open-ended Visual Question Answering Guided by World Knowledge [155.81786738036578]
  Open-ended Visual Question Answering (VQA) task requires AI models to jointly reason over visual and natural language inputs. Pre-trained Language Models (PLM) such as GPT-3 have been applied to the task and shown to be powerful world knowledge sources. We propose RASO: a new VQA pipeline that deploys a generate-then-select strategy guided by world knowledge.
  arXiv  Detail & Related papers  (2023-05-30T08:34:13Z)
• Few-Shot Data Synthesis for Open Domain Multi-Hop Question Answering [40.86455734818704]
  Few-shot learning for open domain multi-hop question answering typically relies on the incontext learning capability of large language models. We propose a data synthesis framework for multi-hop question answering that requires less than 10 human annotated question answer pairs.
  arXiv  Detail & Related papers  (2023-05-23T04:57:31Z)
• AnnoLLM: Making Large Language Models to Be Better Crowdsourced Annotators [98.11286353828525]
  GPT-3.5 series models have demonstrated remarkable few-shot and zero-shot ability across various NLP tasks. We propose AnnoLLM, which adopts a two-step approach, explain-then-annotate. We build the first conversation-based information retrieval dataset employing AnnoLLM.
  arXiv  Detail & Related papers  (2023-03-29T17:03:21Z)
• Self-Prompting Large Language Models for Zero-Shot Open-Domain QA [67.08732962244301]
  Open-Domain Question Answering (ODQA) aims to answer questions without explicitly providing background documents. This task becomes notably challenging in a zero-shot setting where no data is available to train tailored retrieval-reader models. We propose a Self-Prompting framework to explicitly utilize the massive knowledge encoded in the parameters of Large Language Models.
  arXiv  Detail & Related papers  (2022-12-16T18:23:43Z)
• An Empirical Study on Few-shot Knowledge Probing for Pretrained Language Models [54.74525882974022]
  We show that few-shot examples can strongly boost the probing performance for both 1-hop and 2-hop relations. In particular, we find that a simple-yet-effective approach of finetuning the bias vectors in the model outperforms existing prompt-engineering methods.
  arXiv  Detail & Related papers  (2021-09-06T23:29:36Z)
• WARP: Word-level Adversarial ReProgramming [13.08689221166729]
  In many applications it is preferable to tune much smaller sets of parameters, so that the majority of parameters can be shared across multiple tasks. We present an alternative approach based on adversarial reprogramming, which extends earlier work on automatic prompt generation. We show that this approach outperforms other methods with a similar number of trainable parameters on SST-2 and MNLI datasets.
  arXiv  Detail & Related papers  (2021-01-01T00:41:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.