Learning and Unlearning of Fabricated Knowledge in Language Models

• URL: http://arxiv.org/abs/2410.21750v1
• Date: Tue, 29 Oct 2024 05:33:14 GMT
• Title: Learning and Unlearning of Fabricated Knowledge in Language Models
• Authors: Chen Sun, Nolan Andrew Miller, Andrey Zhmoginov, Max Vladymyrov, Mark Sandler,
• Abstract summary: We show that facts that conflict with common knowledge are remembered for tens of thousands of training steps. We show that impacts of knowledge-conflicting facts in LMs, though they can be long lasting, can be largely erased by novel application of multi-step sparse updates.
• Score: 16.971082623826263
• License:
• Abstract: What happens when a new piece of knowledge is introduced into the training data and how long does it last while a large language model (LM) continues to train? We investigate this question by injecting facts into LMs from a new probing dataset, "Outlandish", which is designed to permit the testing of a spectrum of different fact types. When studying how robust these memories are, there appears to be a sweet spot in the spectrum of fact novelty between consistency with world knowledge and total randomness, where the injected memory is the most enduring. Specifically we show that facts that conflict with common knowledge are remembered for tens of thousands of training steps, while prompts not conflicting with common knowledge (mundane), as well as scrambled prompts (randomly jumbled) are both forgotten much more rapidly. Further, knowledge-conflicting facts can "prime'' how the language model hallucinates on logically unrelated prompts, showing their propensity for non-target generalization, while both mundane and randomly jumbled facts prime significantly less. Finally, we show that impacts of knowledge-conflicting facts in LMs, though they can be long lasting, can be largely erased by novel application of multi-step sparse updates, even while the training ability of the model is preserved. As such, this very simple procedure has direct implications for mitigating the effects of data poisoning in training.

Related papers

• Co-occurrence is not Factual Association in Language Models [19.708303468664088]
  We show that language models are biased to learn word co-occurrence statistics instead of true factual associations. We propose two strategies to improve the learning of factual associations in language models.
  arXiv  Detail & Related papers  (2024-09-21T08:13:16Z)
• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? [33.702498916775426]
  We study the impact of new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. We demonstrate that large language models struggle to acquire new factual knowledge through fine-tuning. As the examples with new knowledge are eventually learned, they linearly increase the model's tendency to hallucinate.
  arXiv  Detail & Related papers  (2024-05-09T17:00:22Z)
• KnowTuning: Knowledge-aware Fine-tuning for Large Language Models [83.5849717262019]
  We propose a knowledge-aware fine-tuning (KnowTuning) method to improve fine-grained and coarse-grained knowledge awareness of LLMs. KnowTuning generates more facts with less factual error rate under fine-grained facts evaluation.
  arXiv  Detail & Related papers  (2024-02-17T02:54:32Z)
• Do Large Language Models Know about Facts? [60.501902866946]
  Large language models (LLMs) have recently driven striking performance improvements across a range of natural language processing tasks. We aim to evaluate the extent and scope of factual knowledge within LLMs by designing the benchmark Pinocchio. Pinocchio contains 20K diverse factual questions that span different sources, timelines, domains, regions, and languages.
  arXiv  Detail & Related papers  (2023-10-08T14:26:55Z)
• Mitigating Temporal Misalignment by Discarding Outdated Facts [58.620269228776294]
  Large language models are often used under temporal misalignment, tasked with answering questions about the present. We propose fact duration prediction: the task of predicting how long a given fact will remain true. Our data and code are released publicly at https://github.com/mikejqzhang/mitigating_misalignment.
  arXiv  Detail & Related papers  (2023-05-24T07:30:08Z)
• Investigating Forgetting in Pre-Trained Representations Through Continual Learning [51.30807066570425]
  We study the effect of representation forgetting on the generality of pre-trained language models. We find that the generality is destructed in various pre-trained LMs, and syntactic and semantic knowledge is forgotten through continual learning.
  arXiv  Detail & Related papers  (2023-05-10T08:27:59Z)
• Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge [72.63368052592004]
  We study LMs' abilities to make inferences based on injected facts (or propagate those facts) We find that existing methods for updating knowledge show little propagation of injected knowledge. Yet, prepending entity definitions in an LM's context improves performance across all settings.
  arXiv  Detail & Related papers  (2023-05-02T17:59:46Z)
• Probing Across Time: What Does RoBERTa Know and When? [70.20775905353794]
  We show that linguistic knowledge is acquired fast, stably, and robustly across domains. Facts and commonsense are slower and more domain-sensitive. We believe that probing-across-time analyses can help researchers understand the complex, intermingled learning that these models undergo and guide us toward more efficient approaches that accomplish necessary learning faster.
  arXiv  Detail & Related papers  (2021-04-16T04:26:39Z)
• Facts as Experts: Adaptable and Interpretable Neural Memory over Symbolic Knowledge [38.48518306055536]
  We develop a neural language model that includes an explicit interface between symbolically interpretable factual information and subsymbolic neural knowledge. We show that this model dramatically improves performance on two knowledge-intensive question-answering tasks.
  arXiv  Detail & Related papers  (2020-07-02T03:05:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.