BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence

• URL: http://arxiv.org/abs/2402.12174v2
• Date: Thu, 30 May 2024 11:26:58 GMT
• Title: BIDER: Bridging Knowledge Inconsistency for Efficient Retrieval-Augmented LLMs via Key Supporting Evidence
• Authors: Jiajie Jin, Yutao Zhu, Yujia Zhou, Zhicheng Dou,
• Abstract summary: This paper introduces BIDER, an approach that refines retrieval documents into Key Supporting Evidence. We train BIDER by learning from crafting KSE, while maximizing its output to align with LLM's information acquisition preferences. Evaluations show BIDER boosts LLMs' answer quality by 7% while reducing input content length in retrieval documents by 80%, outperforming existing methods.
• Score: 23.55601157586831
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented large language models (LLMs) have demonstrated efficacy in knowledge-intensive tasks such as open-domain QA, addressing inherent challenges in knowledge update and factual inadequacy. However, inconsistencies between retrieval knowledge and the necessary knowledge for LLMs, leading to a decline in LLM's answer quality. This paper introduces BIDER, an approach that refines retrieval documents into Key Supporting Evidence (KSE) through knowledge synthesis, supervised fine-tuning (SFT), and preference alignment. We train BIDER by learning from crafting KSE, while maximizing its output to align with LLM's information acquisition preferences through reinforcement learning. Evaluations across five datasets show BIDER boosts LLMs' answer quality by 7% while reducing input content length in retrieval documents by 80%, outperforming existing methods. The proposed KSE simulation effectively equips LLMs with essential information for accurate question answering.

Related papers

• KBAlign: Efficient Self Adaptation on Specific Knowledge Bases [75.78948575957081]
  Large language models (LLMs) usually rely on retrieval-augmented generation to exploit knowledge materials in an instant manner. We propose KBAlign, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently.
  arXiv  Detail & Related papers  (2024-11-22T08:21:03Z)
• On the Role of Long-tail Knowledge in Retrieval Augmented Large Language Models [33.08049246893537]
  Retrieval augmented generation (RAG) exhibits outstanding performance in promoting the knowledge capabilities of large language models (LLMs) We propose a simple but effective long-tail knowledge detection method for LLMs. Our method achieves over 4x speedup in average inference time and consistent performance improvement in downstream tasks.
  arXiv  Detail & Related papers  (2024-06-24T07:17:59Z)
• Supportiveness-based Knowledge Rewriting for Retrieval-augmented Language Modeling [65.72918416258219]
  Supportiveness-based Knowledge Rewriting (SKR) is a robust and pluggable knowledge rewriter inherently optimized for LLM generation. Based on knowledge supportiveness, we first design a training data curation strategy for our rewriter model. We then introduce the direct preference optimization (DPO) algorithm to align the generated rewrites to optimal supportiveness.
  arXiv  Detail & Related papers  (2024-06-12T11:52:35Z)
• Evidence-Focused Fact Summarization for Knowledge-Augmented Zero-Shot Question Answering [14.389264346634507]
  We propose EFSum, an Evidence-focused Fact Summarization framework for enhanced Quesetion Answering (QA) performance. Our experiments show that EFSum improves LLM's zero-shot QA performance.
  arXiv  Detail & Related papers  (2024-03-05T13:43:58Z)
• Small Models, Big Insights: Leveraging Slim Proxy Models To Decide When and What to Retrieve for LLMs [60.40396361115776]
  This paper introduces a novel collaborative approach, namely SlimPLM, that detects missing knowledge in large language models (LLMs) with a slim proxy model. We employ a proxy model which has far fewer parameters, and take its answers as answers. Heuristic answers are then utilized to predict the knowledge required to answer the user question, as well as the known and unknown knowledge within the LLM.
  arXiv  Detail & Related papers  (2024-02-19T11:11:08Z)
• TRACE: A Comprehensive Benchmark for Continual Learning in Large Language Models [52.734140807634624]
  Aligned large language models (LLMs) demonstrate exceptional capabilities in task-solving, following instructions, and ensuring safety. Existing continual learning benchmarks lack sufficient challenge for leading aligned LLMs. We introduce TRACE, a novel benchmark designed to evaluate continual learning in LLMs.
  arXiv  Detail & Related papers  (2023-10-10T16:38:49Z)
• Self-Knowledge Guided Retrieval Augmentation for Large Language Models [59.771098292611846]
  Large language models (LLMs) have shown superior performance without task-specific fine-tuning. Retrieval-based methods can offer non-parametric world knowledge and improve the performance on tasks such as question answering. Self-Knowledge guided Retrieval augmentation (SKR) is a simple yet effective method which can let LLMs refer to the questions they have previously encountered.
  arXiv  Detail & Related papers  (2023-10-08T04:22:33Z)
• Knowledgeable In-Context Tuning: Exploring and Exploiting Factual Knowledge for In-Context Learning [37.22349652230841]
  Large language models (LLMs) enable in-context learning (ICL) by conditioning on a few labeled training examples as a text-based prompt. In this paper, we demonstrate that factual knowledge is imperative for the performance of ICL in three core facets. We introduce a novel Knowledgeable In-Context Tuning (KICT) framework to further improve the performance of ICL.
  arXiv  Detail & Related papers  (2023-09-26T09:06:39Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [109.8527403904657]
  We show that large language models (LLMs) possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We propose a simple method to dynamically utilize supporting documents with our judgement strategy.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.