Self-Knowledge Guided Retrieval Augmentation for Large Language Models

• URL: http://arxiv.org/abs/2310.05002v1
• Date: Sun, 8 Oct 2023 04:22:33 GMT
• Title: Self-Knowledge Guided Retrieval Augmentation for Large Language Models
• Authors: Yile Wang, Peng Li, Maosong Sun, Yang Liu
• Abstract summary: Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. Self-Knowledge guided Retrieval augmentation (SKR) is a simple yet effective method which can let LLMs refer to the questions they have previously encountered.
• Score: 59.771098292611846
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Despite the success, the knowledge stored in the parameters of LLMs could still be incomplete and difficult to update due to the computational costs. As complementary, retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. However, we find that the retrieved knowledge does not always help and even has a negative impact on original responses occasionally. To better make use of both internal knowledge and external world knowledge, we investigate eliciting the model's ability to recognize what they know and do not know (which is also called self-knowledge) and propose Self-Knowledge guided Retrieval augmentation (SKR), a simple yet effective method which can let LLMs refer to the questions they have previously encountered and adaptively call for external resources when dealing with new questions. We evaluate SKR on multiple datasets and demonstrate that it outperforms chain-of-thought based and fully retrieval-based methods by using either InstructGPT or ChatGPT.

Related papers

• KBAlign: Efficient Self Adaptation on Specific Knowledge Bases [75.78948575957081]
  Large language models (LLMs) usually rely on retrieval-augmented generation to exploit knowledge materials in an instant manner. We propose KBAlign, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently.
  arXiv  Detail & Related papers  (2024-11-22T08:21:03Z)
• Exploring Knowledge Boundaries in Large Language Models for Retrieval Judgment [56.87031484108484]
  Large Language Models (LLMs) are increasingly recognized for their practical applications. Retrieval-Augmented Generation (RAG) tackles this challenge and has shown a significant impact on LLMs. By minimizing retrieval requests that yield neutral or harmful results, we can effectively reduce both time and computational costs.
  arXiv  Detail & Related papers  (2024-11-09T15:12:28Z)
• On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models [33.08049246893537]
  Retrieval augmented generation (RAG) exhibits outstanding performance in promoting the knowledge capabilities of large language models (LLMs) We propose a simple but effective long-tail knowledge detection method for LLMs. Our method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks.
  arXiv  Detail & Related papers  (2024-06-24T07:17:59Z)
• Knowledge Unlearning for LLMs: Tasks, Methods, and Challenges [11.228131492745842]
  Large language models (LLMs) have spurred a new research paradigm in natural language processing. Despite their excellent capability in knowledge-based question answering and reasoning, their potential to retain faulty or even harmful knowledge poses risks of malicious application. Knowledge unlearning, derived from analogous studies on machine unlearning, presents a promising avenue to address this concern.
  arXiv  Detail & Related papers  (2023-11-27T12:37:51Z)
• RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge [69.79676144482792]
  This study aims to evaluate the ability of LLMs to distinguish reliable information from external knowledge. Our benchmark consists of two tasks, Question Answering and Text Generation, and for each task, we provide models with a context containing counterfactual information.
  arXiv  Detail & Related papers  (2023-11-14T13:24:19Z)
• Thrust: Adaptively Propels Large Language Models with External Knowledge [58.72867916604562]
  Large-scale pre-trained language models (PTLMs) are shown to encode rich knowledge in their model parameters. The inherent knowledge in PTLMs can be opaque or static, making external knowledge necessary. We propose the instance-level adaptive propulsion of external knowledge (IAPEK), where we only conduct the retrieval when necessary.
  arXiv  Detail & Related papers  (2023-07-19T20:16:46Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.