Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and
Mitigating Knowledge Conflicts in Language Models
- URL: http://arxiv.org/abs/2402.18154v1
- Date: Wed, 28 Feb 2024 08:34:41 GMT
- Title: Cutting Off the Head Ends the Conflict: A Mechanism for Interpreting and
Mitigating Knowledge Conflicts in Language Models
- Authors: Zhuoran Jin, Pengfei Cao, Hongbang Yuan, Yubo Chen, Jiexin Xu, Huaijun
Li, Xiaojian Jiang, Kang Liu, Jun Zhao
- Abstract summary: Internal memory and external context inevitably clash, leading to knowledge conflicts within language models (LMs)
We propose a novel method called PatH PatcHing (PH3), which can efficiently mitigate knowledge conflicts by pruning conflicting attention heads without updating model parameters.
- Score: 18.2500350157507
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Recently, retrieval augmentation and tool augmentation have demonstrated a
remarkable capability to expand the internal memory boundaries of language
models (LMs) by providing external context. However, internal memory and
external context inevitably clash, leading to knowledge conflicts within LMs.
In this paper, we aim to interpret the mechanism of knowledge conflicts through
the lens of information flow, and then mitigate conflicts by precise
interventions at the pivotal point. We find there are some attention heads with
opposite effects in the later layers, where memory heads can recall knowledge
from internal memory, and context heads can retrieve knowledge from external
context. Moreover, we reveal that the pivotal point at which knowledge
conflicts emerge in LMs is the integration of inconsistent information flows by
memory heads and context heads. Inspired by the insights, we propose a novel
method called Pruning Head via PatH PatcHing (PH3), which can efficiently
mitigate knowledge conflicts by pruning conflicting attention heads without
updating model parameters. PH3 can flexibly control eight LMs to use internal
memory ($\uparrow$ 44.0%) or external context ($\uparrow$ 38.5%). Moreover, PH3
can also improve the performance of LMs on open-domain QA tasks. We also
conduct extensive experiments to demonstrate the cross-model, cross-relation,
and cross-format generalization of our method.
Related papers
- From Internal Conflict to Contextual Adaptation of Language Models [42.776896363518844]
Studies indicate that LMs often ignore the provided context as it can conflict with the pre-existing LM's memory learned during pre-training.
We introduce the DYNAMICQA dataset, which includes facts with a temporal dynamic nature.
Our experiments reveal that static facts, which are unlikely to change, are more easily updated with additional context.
arXiv Detail & Related papers (2024-07-24T06:06:07Z) - LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements [59.71218039095155]
Task of reading comprehension (RC) provides a primary means to assess language models' natural language understanding (NLU) capabilities.
If the context aligns with the models' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from internal information.
To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities.
arXiv Detail & Related papers (2024-04-09T13:08:56Z) - Untangle the KNOT: Interweaving Conflicting Knowledge and Reasoning Skills in Large Language Models [51.72963030032491]
Knowledge documents for large language models (LLMs) may conflict with the memory of LLMs due to outdated or incorrect knowledge.
We construct a new dataset, dubbed KNOT, for knowledge conflict resolution examination in the form of question answering.
arXiv Detail & Related papers (2024-04-04T16:40:11Z) - Tuning-Free Accountable Intervention for LLM Deployment -- A
Metacognitive Approach [55.613461060997004]
Large Language Models (LLMs) have catalyzed transformative advances across a spectrum of natural language processing tasks.
We propose an innovative textitmetacognitive approach, dubbed textbfCLEAR, to equip LLMs with capabilities for self-aware error identification and correction.
arXiv Detail & Related papers (2024-03-08T19:18:53Z) - Discerning and Resolving Knowledge Conflicts through Adaptive Decoding with Contextual Information-Entropy Constraint [20.543282448771336]
We propose an adaptive decoding method to discern whether the knowledge conflicts occur and resolve them.
Experiments show that COIECD exhibits strong performance and robustness over knowledge conflicts in realistic datasets.
arXiv Detail & Related papers (2024-02-19T07:10:30Z) - Retrieve Only When It Needs: Adaptive Retrieval Augmentation for
Hallucination Mitigation in Large Language Models [73.93616728895401]
Hallucinations pose a significant challenge for the practical implementation of large language models (LLMs)
We present Rowen, a novel approach that enhances LLMs with a selective retrieval augmentation process tailored to address hallucinations.
arXiv Detail & Related papers (2024-02-16T11:55:40Z) - A Framework for Inference Inspired by Human Memory Mechanisms [9.408704431898279]
We propose a PMI framework that consists of perception, memory and inference components.
The memory module comprises working and long-term memory, with the latter endowed with a higher-order structure to retain extensive and complex relational knowledge and experience.
We apply our PMI to improve prevailing Transformers and CNN models on question-answering tasks like bAbI-20k and Sort-of-CLEVR datasets.
arXiv Detail & Related papers (2023-10-01T08:12:55Z) - Augmenting LLMs with Knowledge: A survey on hallucination prevention [0.0]
This survey delves into the realm of language models (LMs) augmented with the ability to tap into external knowledge sources.
While adhering to the standard objective of predicting missing tokens, these augmented LMs leverage diverse, possibly non-parametric external modules.
arXiv Detail & Related papers (2023-09-28T14:09:58Z) - Improving Open Information Extraction with Large Language Models: A
Study on Demonstration Uncertainty [52.72790059506241]
Open Information Extraction (OIE) task aims at extracting structured facts from unstructured text.
Despite the potential of large language models (LLMs) like ChatGPT as a general task solver, they lag behind state-of-the-art (supervised) methods in OIE tasks.
arXiv Detail & Related papers (2023-09-07T01:35:24Z) - Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large
Language Models in Knowledge Conflicts [21.34852490049787]
We present the first comprehensive and controlled investigation into the behavior of large language models (LLMs) when encountering knowledge conflicts.
We find that LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory.
On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information consistent with their parametric memory.
arXiv Detail & Related papers (2023-05-22T17:57:41Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.