Tug-of-War Between Knowledge: Exploring and Resolving Knowledge
Conflicts in Retrieval-Augmented Language Models
- URL: http://arxiv.org/abs/2402.14409v1
- Date: Thu, 22 Feb 2024 09:51:08 GMT
- Title: Tug-of-War Between Knowledge: Exploring and Resolving Knowledge
Conflicts in Retrieval-Augmented Language Models
- Authors: Zhuoran Jin, Pengfei Cao, Yubo Chen, Kang Liu, Xiaojian Jiang, Jiexin
Xu, Qiuxia Li, Jun Zhao
- Abstract summary: Retrieval-augmented language models (RALMs) have demonstrated significant potential in refining and expanding their internal memory.
Knowledge conflicts can ensnare RALMs in a tug-of-war between knowledge, limiting their practical applicability.
We propose a method called Conflict-Disentangle Contrastive Decoding (CD2) to better calibrate the model's confidence.
- Score: 18.82042974470535
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Retrieval-augmented language models (RALMs) have demonstrated significant
potential in refining and expanding their internal memory by retrieving
evidence from external sources. However, RALMs will inevitably encounter
knowledge conflicts when integrating their internal memory with external
sources. Knowledge conflicts can ensnare RALMs in a tug-of-war between
knowledge, limiting their practical applicability. In this paper, we focus on
exploring and resolving knowledge conflicts in RALMs. First, we present an
evaluation framework for assessing knowledge conflicts across various
dimensions. Then, we investigate the behavior and preference of RALMs from the
following two perspectives: (1) Conflicts between internal memory and external
sources: We find that stronger RALMs emerge with the Dunning-Kruger effect,
persistently favoring their faulty internal memory even when correct evidence
is provided. Besides, RALMs exhibit an availability bias towards common
knowledge; (2) Conflicts between truthful, irrelevant and misleading evidence:
We reveal that RALMs follow the principle of majority rule, leaning towards
placing trust in evidence that appears more frequently. Moreover, we find that
RALMs exhibit confirmation bias, and are more willing to choose evidence that
is consistent with their internal memory. To solve the challenge of knowledge
conflicts, we propose a method called Conflict-Disentangle Contrastive Decoding
(CD2) to better calibrate the model's confidence. Experimental results
demonstrate that our CD2 can effectively resolve knowledge conflicts in RALMs.
Related papers
- From Internal Conflict to Contextual Adaptation of Language Models [42.776896363518844]
Studies indicate that LMs often ignore the provided context as it can conflict with the pre-existing LM's memory learned during pre-training.
We introduce the DYNAMICQA dataset, which includes facts with a temporal dynamic nature.
Our experiments reveal that static facts, which are unlikely to change, are more easily updated with additional context.
arXiv Detail & Related papers (2024-07-24T06:06:07Z) - LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements [59.71218039095155]
Task of reading comprehension (RC) provides a primary means to assess language models' natural language understanding (NLU) capabilities.
If the context aligns with the models' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from internal information.
To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities.
arXiv Detail & Related papers (2024-04-09T13:08:56Z) - Untangle the KNOT: Interweaving Conflicting Knowledge and Reasoning Skills in Large Language Models [51.72963030032491]
Knowledge documents for large language models (LLMs) may conflict with the memory of LLMs due to outdated or incorrect knowledge.
We construct a new dataset, dubbed KNOT, for knowledge conflict resolution examination in the form of question answering.
arXiv Detail & Related papers (2024-04-04T16:40:11Z) - Knowledge Conflicts for LLMs: A Survey [24.731074825915833]
Survey focuses on three categories of knowledge conflicts: context-memory, inter-context, and intra-memory conflict.
These conflicts can significantly impact the trustworthiness and performance of large language models.
arXiv Detail & Related papers (2024-03-13T08:02:23Z) - TrustScore: Reference-Free Evaluation of LLM Response Trustworthiness [58.721012475577716]
Large Language Models (LLMs) have demonstrated impressive capabilities across various domains, prompting a surge in their practical applications.
This paper introduces TrustScore, a framework based on the concept of Behavioral Consistency, which evaluates whether an LLMs response aligns with its intrinsic knowledge.
arXiv Detail & Related papers (2024-02-19T21:12:14Z) - Resolving Knowledge Conflicts in Large Language Models [48.92369530853202]
Large language models (LLMs) often encounter knowledge conflicts.
We ask what are the desiderata for LLMs when a knowledge conflict arises and whether existing LLMs fulfill them.
We introduce KNOWLEDGE CONFLICT, an evaluation framework for simulating contextual knowledge conflicts.
arXiv Detail & Related papers (2023-10-02T06:57:45Z) - Getting Sick After Seeing a Doctor? Diagnosing and Mitigating Knowledge Conflicts in Event Temporal Reasoning [87.92209048521153]
Event temporal reasoning aims at identifying the temporal relations between two or more events from narratives.
Knowledge conflicts arise when there is a mismatch between the actual temporal relations of events in the context and the prior knowledge or biases learned by the model.
arXiv Detail & Related papers (2023-05-24T10:04:06Z) - Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large
Language Models in Knowledge Conflicts [21.34852490049787]
We present the first comprehensive and controlled investigation into the behavior of large language models (LLMs) when encountering knowledge conflicts.
We find that LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory.
On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information consistent with their parametric memory.
arXiv Detail & Related papers (2023-05-22T17:57:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.