Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering

• URL: http://arxiv.org/abs/2410.15999v2
• Date: Fri, 25 Oct 2024 14:17:28 GMT
• Title: Steering Knowledge Selection Behaviours in LLMs via SAE-Based Representation Engineering
• Authors: Yu Zhao, Alessio Devoto, Giwon Hong, Xiaotang Du, Aryo Pradipta Gema, Hongru Wang, Xuanli He, Kam-Fai Wong, Pasquale Minervini,
• Abstract summary: Large language models (LLMs) can store a significant amount of factual knowledge in their parameters. LLMs can internally register the signals of knowledge conflict at mid-layers. We propose textscSpARE, a representation engineering method that uses pre-trained sparse auto-encoders.
• Score: 23.96385393039587
• License:
• Abstract: Large language models (LLMs) can store a significant amount of factual knowledge in their parameters. However, their parametric knowledge may conflict with the information provided in the context -- this phenomenon, known as \emph{context-memory knowledge conflicts}, can lead to undesirable model behaviour, such as reliance on outdated or incorrect information. Analysing the internal activations of LLMs, we find that they can internally register the signals of knowledge conflict at mid-layers. Such signals allow us to detect whether a knowledge conflict occurs and use \emph{inference-time} intervention strategies to resolve it. In this work, we propose \textsc{SpARE}, a \emph{training-free} representation engineering method that uses pre-trained sparse auto-encoders (SAEs) to control the knowledge selection behaviour of LLMs. \textsc{SpARE} identifies the functional features that control the knowledge selection behaviours and applies them to edit the internal activations of LLMs at inference time. Our experimental results show that \textsc{SpARE} can effectively control the usage of either knowledge source to resolve knowledge conflict in open-domain question-answering tasks, surpassing existing representation engineering methods ($+10\%$) as well as contrastive decoding methods ($+15\%$).

Related papers

• Analysing the Residual Stream of Language Models Under Knowledge Conflicts [23.96385393039587]
  Large language models (LLMs) can store a significant amount of factual knowledge in their parameters. However, their parametric knowledge may conflict with the information provided in the context. This can lead to undesirable model behaviour, such as reliance on outdated or incorrect information.
  arXiv  Detail & Related papers  (2024-10-21T15:12:51Z)
• Answer When Needed, Forget When Not: Language Models Pretend to Forget via In-Context Knowledge Unlearning [26.861562920084264]
  Large language models (LLMs) are applied across diverse domains. We propose a novel method termed in-context knowledge unlearning'' Our method fine-tunes pre-trained LLMs to enable prompt unlearning of target knowledge within the context.
  arXiv  Detail & Related papers  (2024-10-01T04:13:25Z)
• LLMs' Reading Comprehension Is Affected by Parametric Knowledge and Struggles with Hypothetical Statements [59.71218039095155]
  Task of reading comprehension (RC) provides a primary means to assess language models' natural language understanding (NLU) capabilities. If the context aligns with the models' internal knowledge, it is hard to discern whether the models' answers stem from context comprehension or from internal information. To address this issue, we suggest to use RC on imaginary data, based on fictitious facts and entities.
  arXiv  Detail & Related papers  (2024-04-09T13:08:56Z)
• Untangle the KNOT: Interweaving Conflicting Knowledge and Reasoning Skills in Large Language Models [51.72963030032491]
  Knowledge documents for large language models (LLMs) may conflict with the memory of LLMs due to outdated or incorrect knowledge. We construct a new dataset, dubbed KNOT, for knowledge conflict resolution examination in the form of question answering.
  arXiv  Detail & Related papers  (2024-04-04T16:40:11Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• ActiveRAG: Autonomously Knowledge Assimilation and Accommodation through Retrieval-Augmented Agents [49.30553350788524]
  Retrieval-Augmented Generation (RAG) enables Large Language Models (LLMs) to leverage external knowledge. Existing RAG models often treat LLMs as passive recipients of information. We introduce ActiveRAG, a multi-agent framework that mimics human learning behavior.
  arXiv  Detail & Related papers  (2024-02-21T06:04:53Z)
• RECALL: A Benchmark for LLMs Robustness against External Counterfactual Knowledge [69.79676144482792]
  This study aims to evaluate the ability of LLMs to distinguish reliable information from external knowledge. Our benchmark consists of two tasks, Question Answering and Text Generation, and for each task, we provide models with a context containing counterfactual information.
  arXiv  Detail & Related papers  (2023-11-14T13:24:19Z)
• "Merge Conflicts!" Exploring the Impacts of External Distractors to Parametric Knowledge Graphs [15.660128743249611]
  Large language models (LLMs) acquire extensive knowledge during pre-training, known as their parametric knowledge. LLMs inevitably require external knowledge during their interactions with users. This raises a crucial question: How will LLMs respond when external knowledge interferes with their parametric knowledge?
  arXiv  Detail & Related papers  (2023-09-15T17:47:59Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.