Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models

• URL: http://arxiv.org/abs/2402.14409v1
• Date: Thu, 22 Feb 2024 09:51:08 GMT
• Title: Tug-of-War Between Knowledge: Exploring and Resolving Knowledge Conflicts in Retrieval-Augmented Language Models
• Authors: Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin Xu, Qiuxia Li, Jun Zhao
• Abstract summary: Retrieval-augmented language models (RALMs) have demonstrated significant potential in refining and expanding their internal memory. Knowledge conflicts can ensnare RALMs in a tug-of-war between knowledge, limiting their practical applicability. We propose a method called Conflict-Disentangle Contrastive Decoding (CD2) to better calibrate the model's confidence.
• Score: 18.82042974470535
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Retrieval-augmented language models (RALMs) have demonstrated significant potential in refining and expanding their internal memory by retrieving evidence from external sources. However, RALMs will inevitably encounter knowledge conflicts when integrating their internal memory with external sources. Knowledge conflicts can ensnare RALMs in a tug-of-war between knowledge, limiting their practical applicability. In this paper, we focus on exploring and resolving knowledge conflicts in RALMs. First, we present an evaluation framework for assessing knowledge conflicts across various dimensions. Then, we investigate the behavior and preference of RALMs from the following two perspectives: (1) Conflicts between internal memory and external sources: We find that stronger RALMs emerge with the Dunning-Kruger effect, persistently favoring their faulty internal memory even when correct evidence is provided. Besides, RALMs exhibit an availability bias towards common knowledge; (2) Conflicts between truthful, irrelevant and misleading evidence: We reveal that RALMs follow the principle of majority rule, leaning towards placing trust in evidence that appears more frequently. Moreover, we find that RALMs exhibit confirmation bias, and are more willing to choose evidence that is consistent with their internal memory. To solve the challenge of knowledge conflicts, we propose a method called Conflict-Disentangle Contrastive Decoding (CD2) to better calibrate the model's confidence. Experimental results demonstrate that our CD2 can effectively resolve knowledge conflicts in RALMs.

Related papers

• When Seeing Overrides Knowing: Disentangling Knowledge Conflicts in Vision-Language Models [13.390492503308792]
  We analyze the mechanisms that Vision-language models (VLMs) use to resolve cross-modal conflicts.<n>We localize with logit inspection a small set of heads that control the conflict.<n>We show that attention from such heads pinpoints localized image regions driving visual overrides, outperforming gradient-based attribution in precision.
  arXiv  Detail & Related papers  (2025-07-18T12:42:30Z)
• Conflicts in Texts: Data, Implications and Challenges [58.03478157713084]
  Conflicts could reflect the complexity of situations, changes that need to be explained and dealt with, difficulties in data annotation, and mistakes in generated outputs.<n>This survey categorizes these conflicts into three key areas: (1) natural texts on the web, where factual inconsistencies, subjective biases, and multiple perspectives introduce contradictions; (2) human-annotated data, where annotator disagreements, mistakes, and societal biases impact model training; and (3) model interactions, where hallucinations and knowledge conflicts emerge during deployment.<n>We highlight key challenges and future directions for developing conflict-aware NLP systems that can reason over and reconcile conflicting information more effectively
  arXiv  Detail & Related papers  (2025-04-28T04:24:01Z)
• PIP-KAG: Mitigating Knowledge Conflicts in Knowledge-Augmented Generation via Parametric Pruning [92.07119924043461]
  Knowledge-Augmented Generation (KAG) has shown great promise in updating the internal memory of Large Language Models (LLMs) Current approaches to mitigating these conflicts mainly focus on improving external knowledge utilization. We propose a ParametrIc Pruning-based Knowledge-Augmented Generation (PIP-KAG) approach, which prunes internal knowledge of LLMs.
  arXiv  Detail & Related papers  (2025-02-21T15:50:41Z)
• Understanding the Interplay between Parametric and Contextual Knowledge for Large Language Models [85.13298925375692]
  Large language models (LLMs) encode vast amounts of knowledge during pre-training. LLMs can be enhanced by incorporating contextual knowledge (CK) Can LLMs effectively integrate their internal PK with external CK to solve complex problems?
  arXiv  Detail & Related papers  (2024-10-10T23:09:08Z)
• ECon: On the Detection and Resolution of Evidence Conflicts [56.89209046429291]
  The rise of large language models (LLMs) has significantly influenced the quality of information in decision-making systems. This study introduces a method for generating diverse, validated evidence conflicts to simulate real-world misinformation scenarios.
  arXiv  Detail & Related papers  (2024-10-05T07:41:17Z)
• ConflictBank: A Benchmark for Evaluating the Influence of Knowledge Conflicts in LLM [36.332500824079844]
  Large language models (LLMs) have achieved impressive advancements across numerous disciplines, yet the critical issue of knowledge conflicts has rarely been studied. We present ConflictBank, the first comprehensive benchmark developed to evaluate knowledge conflicts from three aspects. Our investigation delves into four model families and twelve LLM instances, meticulously analyzing conflicts stemming from misinformation, temporal discrepancies, and semantic divergences.
  arXiv  Detail & Related papers  (2024-08-22T02:33:13Z)
• DYNAMICQA: Tracing Internal Knowledge Conflicts in Language Models [42.776896363518844]
  We study the effect of intra-memory conflict on an LM's ability to accept relevant context. We utilize two knowledge conflict measures and a novel dataset containing inherently conflicting data, DynamicQA. We verify that LMs exhibit a greater degree of intra-memory conflict with dynamic facts compared to facts that have a single truth value.
  arXiv  Detail & Related papers  (2024-07-24T06:06:07Z)
• Untangle the KNOT: Interweaving Conflicting Knowledge and Reasoning Skills in Large Language Models [51.72963030032491]
  Knowledge documents for large language models (LLMs) may conflict with the memory of LLMs due to outdated or incorrect knowledge. We construct a new dataset, dubbed KNOT, for knowledge conflict resolution examination in the form of question answering.
  arXiv  Detail & Related papers  (2024-04-04T16:40:11Z)
• Knowledge Conflicts for LLMs: A Survey [24.731074825915833]
  Survey focuses on three categories of knowledge conflicts: context-memory, inter-context, and intra-memory conflict. These conflicts can significantly impact the trustworthiness and performance of large language models.
  arXiv  Detail & Related papers  (2024-03-13T08:02:23Z)
• Resolving Knowledge Conflicts in Large Language Models [46.903549751371415]
  Large language models (LLMs) often encounter knowledge conflicts. We ask what are the desiderata for LLMs when a knowledge conflict arises and whether existing LLMs fulfill them. We introduce an evaluation framework for simulating contextual knowledge conflicts.
  arXiv  Detail & Related papers  (2023-10-02T06:57:45Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [109.8527403904657]
  We show that large language models (LLMs) possess unwavering confidence in their knowledge and cannot handle the conflict between internal and external knowledge well. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We propose a simple method to dynamically utilize supporting documents with our judgement strategy.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)
• Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning [87.92209048521153]
  Event temporal reasoning aims at identifying the temporal relations between two or more events from narratives. Knowledge conflicts arise when there is a mismatch between the actual temporal relations of events in the context and the prior knowledge or biases learned by the model.
  arXiv  Detail & Related papers  (2023-05-24T10:04:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.