Entropy-Based Decoding for Retrieval-Augmented Large Language Models

• URL: http://arxiv.org/abs/2406.17519v1
• Date: Tue, 25 Jun 2024 12:59:38 GMT
• Title: Entropy-Based Decoding for Retrieval-Augmented Large Language Models
• Authors: Zexuan Qiu, Zijing Ou, Bin Wu, Jingjing Li, Aiwei Liu, Irwin King,
• Abstract summary: Augmenting Large Language Models with retrieved external knowledge has proven effective for improving the factual accuracy of generated responses. We introduce a novel, training-free decoding method guided by entropy considerations to mitigate this issue.
• Score: 43.93281157539377
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Augmenting Large Language Models (LLMs) with retrieved external knowledge has proven effective for improving the factual accuracy of generated responses. Despite their success, retrieval-augmented LLMs still face the distractibility issue, where the generated responses are negatively influenced by noise from both external and internal knowledge sources. In this paper, we introduce a novel, training-free decoding method guided by entropy considerations to mitigate this issue. Our approach utilizes entropy-based document-parallel ensemble decoding to prioritize low-entropy distributions from retrieved documents, thereby enhancing the extraction of relevant information of context. Additionally, it incorporates a contrastive decoding mechanism that contrasts the obtained low-entropy ensemble distribution with the high-entropy distribution derived from the model's internal knowledge across layers, which ensures a greater emphasis on reliable external information. Extensive experiments on open-domain question answering datasets demonstrate the superiority of our method.

Related papers

• Towards Building a Robust Knowledge Intensive Question Answering Model with Large Language Models [4.4849006637642805]
  Presence of noise and errors in retrieved information poses challenges to the robustness of LLMs. To address the issue of model accuracy decline caused by noisy external information, we propose a data augmentation-based fine-tuning method. We have conducted experiments on both existing LLMs and our approach, the results are evaluated by GPT-4.
  arXiv  Detail & Related papers  (2024-09-09T07:32:30Z)
• Neural Surface Reconstruction from Sparse Views Using Epipolar Geometry [4.659427498118277]
  We present a novel approach, named EpiS, that incorporates Epipolar information into the reconstruction process. Our method aggregates coarse information from the cost volume into Epipolar features extracted from multiple source views. To address the information gaps in sparse conditions, we integrate depth information from monocular depth estimation using global and local regularization techniques.
  arXiv  Detail & Related papers  (2024-06-06T17:47:48Z)
• Compressing Long Context for Enhancing RAG with AMR-based Concept Distillation [17.156915103545728]
  Large Language Models (LLMs) have made significant strides in information acquisition. Retrieval Augmented Generation (RAG) addresses this limitation by incorporating external, non-parametric knowledge. We propose a novel concept-based RAG framework with the Abstract Representation (AMR)-based concept distillation algorithm.
  arXiv  Detail & Related papers  (2024-05-06T00:18:43Z)
• REMEDI: Corrective Transformations for Improved Neural Entropy Estimation [0.7488108981865708]
  We introduce $textttREMEDI$ for efficient and accurate estimation of differential entropy. Our approach demonstrates improvement across a broad spectrum of estimation tasks. It can be naturally extended to information theoretic supervised learning models.
  arXiv  Detail & Related papers  (2024-02-08T14:47:37Z)
• The Risk of Federated Learning to Skew Fine-Tuning Features and Underperform Out-of-Distribution Robustness [50.52507648690234]
  Federated learning has the risk of skewing fine-tuning features and compromising the robustness of the model. We introduce three robustness indicators and conduct experiments across diverse robust datasets. Our approach markedly enhances the robustness across diverse scenarios, encompassing various parameter-efficient fine-tuning methods.
  arXiv  Detail & Related papers  (2024-01-25T09:18:51Z)
• STEERING: Stein Information Directed Exploration for Model-Based Reinforcement Learning [111.75423966239092]
  We propose an exploration incentive in terms of the integral probability metric (IPM) between a current estimate of the transition model and the unknown optimal. Based on KSD, we develop a novel algorithm algo: textbfSTEin information dirtextbfEcted exploration for model-based textbfReinforcement LearntextbfING.
  arXiv  Detail & Related papers  (2023-01-28T00:49:28Z)
• Causality-Aware Local Interpretable Model-Agnostic Explanations [7.412445894287709]
  We propose a novel extension to a widely used local and model-agnostic explainer, which encodes explicit causal relationships within the data surrounding the instance being explained. Our approach overcomes the original method in terms of faithfully replicating the black-box model's mechanism and the consistency and reliability of the generated explanations.
  arXiv  Detail & Related papers  (2022-12-10T10:12:27Z)
• Principled Knowledge Extrapolation with GANs [92.62635018136476]
  We study counterfactual synthesis from a new perspective of knowledge extrapolation. We show that an adversarial game with a closed-form discriminator can be used to address the knowledge extrapolation problem. Our method enjoys both elegant theoretical guarantees and superior performance in many scenarios.
  arXiv  Detail & Related papers  (2022-05-21T08:39:42Z)
• Improving Long Tailed Document-Level Relation Extraction via Easy Relation Augmentation and Contrastive Learning [66.83982926437547]
  We argue that mitigating the long-tailed distribution problem is crucial for DocRE in the real-world scenario. Motivated by the long-tailed distribution problem, we propose an Easy Relation Augmentation(ERA) method for improving DocRE.
  arXiv  Detail & Related papers  (2022-05-21T06:15:11Z)
• SAIS: Supervising and Augmenting Intermediate Steps for Document-Level Relation Extraction [51.27558374091491]
  We propose to explicitly teach the model to capture relevant contexts and entity types by supervising and augmenting intermediate steps (SAIS) for relation extraction. Based on a broad spectrum of carefully designed tasks, our proposed SAIS method not only extracts relations of better quality due to more effective supervision, but also retrieves the corresponding supporting evidence more accurately.
  arXiv  Detail & Related papers  (2021-09-24T17:37:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.