Self-Knowledge Guided Retrieval Augmentation for Large Language Models

• URL: http://arxiv.org/abs/2310.05002v1
• Date: Sun, 8 Oct 2023 04:22:33 GMT
• Title: Self-Knowledge Guided Retrieval Augmentation for Large Language Models
• Authors: Yile Wang, Peng Li, Maosong Sun, Yang Liu
• Abstract summary: Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. Self-Knowledge guided Retrieval augmentation (SKR) is a simple yet effective method which can let LLMs refer to the questions they have previously encountered.
• Score: 59.771098292611846
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs. As complementary, retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally. To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called self-knowledge) and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either InstructGPT or ChatGPT.

Related papers

• LINKED: Eliciting, Filtering and Integrating Knowledge in Large Language Model for Commonsense Reasoning [21.12539851761666]
  Large language models (LLMs) sometimes demonstrate poor performance on knowledge-intensive tasks. We propose a novel method named eliciting, filtering and integrating knowledge in large language model (LINKED) With our comprehensive experiments on two complex commonsense reasoning benchmarks, our method outperforms SOTA baselines (up to 9.0% improvement of accuracy)
  arXiv  Detail & Related papers  (2024-10-12T14:12:22Z)
• Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling [65.72918416258219]
  Supportiveness-based Knowledge Rewriting (SKR) is a robust and pluggable knowledge rewriter inherently optimized for LLM generation. Based on knowledge supportiveness, we first design a training data curation strategy for our rewriter model. We then introduce the direct preference optimization (DPO) algorithm to align the generated rewrites to optimal supportiveness.
  arXiv  Detail & Related papers  (2024-06-12T11:52:35Z)
• Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges [11.228131492745842]
  Large language models (LLMs) have spurred a new research paradigm in natural language processing. Despite their excellent capability in knowledge-based question answering and reasoning, their potential to retain faulty or even harmful knowledge poses risks of malicious application. Knowledge unlearning, derived from analogous studies on machine unlearning, presents a promising avenue to address this concern.
  arXiv  Detail & Related papers  (2023-11-27T12:37:51Z)
• RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge [69.79676144482792]
  This study aims to evaluate the ability of LLMs to distinguish reliable information from external knowledge. Our benchmark consists of two tasks, Question Answering and Text Generation, and for each task, we provide models with a context containing counterfactual information.
  arXiv  Detail & Related papers  (2023-11-14T13:24:19Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [91.30946119104111]
  We show that large language models (LLMs) possess unwavering confidence in their capabilities to respond to questions. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We also find that LLMs have a propensity to rely on the provided retrieval results when formulating answers.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)
• Thrust: Adaptively Propels Large Language Models with External Knowledge [58.72867916604562]
  Large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters. The inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. We propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary.
  arXiv  Detail & Related papers  (2023-07-19T20:16:46Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.