Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts

• URL: http://arxiv.org/abs/2305.13300v4
• Date: Tue, 27 Feb 2024 17:08:49 GMT
• Title: Adaptive Chameleon or Stubborn Sloth: Revealing the Behavior of Large Language Models in Knowledge Conflicts
• Authors: Jian Xie, Kai Zhang, Jiangjie Chen, Renze Lou, Yu Su
• Abstract summary: We present the first comprehensive and controlled investigation into the behavior of large language models (LLMs) when encountering knowledge conflicts. We find that LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory. On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information consistent with their parametric memory.
• Score: 21.34852490049787
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: By providing external information to large language models (LLMs), tool augmentation (including retrieval augmentation) has emerged as a promising solution for addressing the limitations of LLMs' static parametric memory. However, how receptive are LLMs to such external evidence, especially when the evidence conflicts with their parametric memory? We present the first comprehensive and controlled investigation into the behavior of LLMs when encountering knowledge conflicts. We propose a systematic framework to elicit high-quality parametric memory from LLMs and construct the corresponding counter-memory, which enables us to conduct a series of controlled experiments. Our investigation reveals seemingly contradicting behaviors of LLMs. On the one hand, different from prior wisdom, we find that LLMs can be highly receptive to external evidence even when that conflicts with their parametric memory, given that the external evidence is coherent and convincing. On the other hand, LLMs also demonstrate a strong confirmation bias when the external evidence contains some information that is consistent with their parametric memory, despite being presented with conflicting evidence at the same time. These results pose important implications that are worth careful consideration for the further development and deployment of tool- and retrieval-augmented LLMs. Resources are available at https://github.com/OSU-NLP-Group/LLM-Knowledge-Conflict.

Related papers

• Analysing the Residual Stream of Language Models Under Knowledge Conflicts [23.96385393039587]
  Large language models (LLMs) can store a significant amount of factual knowledge in their parameters. However, their parametric knowledge may conflict with the information provided in the context. This can lead to undesirable model behaviour, such as reliance on outdated or incorrect information.
  arXiv  Detail & Related papers  (2024-10-21T15:12:51Z)
• Investigating Context-Faithfulness in Large Language Models: The Roles of Memory Strength and Evidence Style [13.968658352075334]
  We investigate the impact of memory strength and evidence presentation on Large Language Models' receptiveness to external evidence. For questions with high memory strength, LLMs are more likely to rely on internal memory, particularly for larger LLMs such as GPT-4.
  arXiv  Detail & Related papers  (2024-09-17T07:44:06Z)
• Unveiling Factual Recall Behaviors of Large Language Models through Knowledge Neurons [13.266817091775042]
  We investigate whether Large Language Models (LLMs) actively recall or retrieve their internal repositories of factual knowledge when faced with reasoning tasks. We reveal that LLMs fail to harness the critical factual associations under certain circumstances. We assess the effect of Chain-of-Thought (CoT) prompting, a powerful technique for addressing complex reasoning tasks.
  arXiv  Detail & Related papers  (2024-08-06T15:07:08Z)
• Evaluating the External and Parametric Knowledge Fusion of Large Language Models [72.40026897037814]
  We develop a systematic pipeline for data construction and knowledge infusion to simulate knowledge fusion scenarios. Our investigation reveals that enhancing parametric knowledge within LLMs can significantly bolster their capability for knowledge integration. Our findings aim to steer future explorations on harmonizing external and parametric knowledge within LLMs.
  arXiv  Detail & Related papers  (2024-05-29T11:48:27Z)
• Untangle the KNOT: Interweaving Conflicting Knowledge and Reasoning Skills in Large Language Models [51.72963030032491]
  Knowledge documents for large language models (LLMs) may conflict with the memory of LLMs due to outdated or incorrect knowledge. We construct a new dataset, dubbed KNOT, for knowledge conflict resolution examination in the form of question answering.
  arXiv  Detail & Related papers  (2024-04-04T16:40:11Z)
• ActiveRAG: Autonomously Knowledge Assimilation and Accommodation through Retrieval-Augmented Agents [49.30553350788524]
  Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to leverage external knowledge. Existing RAG models often treat LLMs as passive recipients of information. We introduce ActiveRAG, a multi-agent framework that mimics human learning behavior.
  arXiv  Detail & Related papers  (2024-02-21T06:04:53Z)
• When Do LLMs Need Retrieval Augmentation? Mitigating LLMs' Overconfidence Helps Retrieval Augmentation [66.01754585188739]
  Large Language Models (LLMs) have been found to have difficulty knowing they do not possess certain knowledge. Retrieval Augmentation (RA) has been extensively studied to mitigate LLMs' hallucinations. We propose several methods to enhance LLMs' perception of knowledge boundaries and show that they are effective in reducing overconfidence.
  arXiv  Detail & Related papers  (2024-02-18T04:57:19Z)
• Retrieve Only When It Needs: Adaptive Retrieval Augmentation for Hallucination Mitigation in Large Language Models [68.91592125175787]
  Hallucinations pose a significant challenge for the practical implementation of large language models (LLMs) We present Rowen, a novel approach that enhances LLMs with a selective retrieval augmentation process tailored to address hallucinations.
  arXiv  Detail & Related papers  (2024-02-16T11:55:40Z)
• "Merge Conflicts!" Exploring the Impacts of External Distractors to Parametric Knowledge Graphs [15.660128743249611]
  Large language models (LLMs) acquire extensive knowledge during pre-training, known as their parametric knowledge. LLMs inevitably require external knowledge during their interactions with users. This raises a crucial question: How will LLMs respond when external knowledge interferes with their parametric knowledge?
  arXiv  Detail & Related papers  (2023-09-15T17:47:59Z)
• Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation [91.30946119104111]
  We show that large language models (LLMs) possess unwavering confidence in their capabilities to respond to questions. Retrieval augmentation proves to be an effective approach in enhancing LLMs' awareness of knowledge boundaries. We also find that LLMs have a propensity to rely on the provided retrieval results when formulating answers.
  arXiv  Detail & Related papers  (2023-07-20T16:46:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.