Understanding Finetuning for Factual Knowledge Extraction

• URL: http://arxiv.org/abs/2406.14785v1
• Date: Thu, 20 Jun 2024 23:27:06 GMT
• Title: Understanding Finetuning for Factual Knowledge Extraction
• Abstract summary: We show that fine-tuning on lesser-known facts that are poorly stored during pretraining yields significantly worse factuality than fine-tuning on well-known facts, even when all facts are seen during pretraining. Our results shed light on the interaction between pretrained knowledge and finetuning data and demonstrate the importance of taking into account how facts are stored in the pretrained model when fine-tuning for knowledge-intensive tasks.
• Score: 45.398982602347765
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this work, we study the impact of QA fine-tuning data on downstream factuality. We show that fine-tuning on lesser-known facts that are poorly stored during pretraining yields significantly worse factuality than fine-tuning on well-known facts, even when all facts are seen during pretraining. We prove this phenomenon theoretically, showing that training on lesser-known facts can lead the model to ignore subject entity names and instead output a generic plausible response even when the relevant factual knowledge is encoded in the model. On three question answering benchmarks (PopQA, Entity Questions, and MMLU) and two language models (Llama-2-7B and Mistral-7B), we find that (i) finetuning on a completely factual but lesser-known subset of the data deteriorates downstream factuality (5-10%) and (ii) finetuning on a subset of better-known examples matches or outperforms finetuning on the entire dataset. Ultimately, our results shed light on the interaction between pretrained knowledge and finetuning data and demonstrate the importance of taking into account how facts are stored in the pretrained model when fine-tuning for knowledge-intensive tasks.

Related papers

• Popular Knowledge Propagates More Errors in LLM Knowledge Updating [64.62006223921576]
  Updating a language model's knowledge through fine-tuning can induce factual forgetting and new hallucinations.<n>We investigate which facts are most vulnerable to collateral corruption during other updates.<n>We propose Popularity-based Anchoring (PopAnchor), a lightweight rehearsal strategy that preserves a small set of popular facts and reduces forgetting.
  arXiv  Detail & Related papers  (2026-09-08T00:23:27Z)
• From Style to Facts: Mapping the Boundaries of Knowledge Injection with Finetuning [40.141932069582204]
  Finetuning provides a scalable and cost-effective means of customizing language models for specific tasks or response styles. In contrast, the conventional wisdom is that injecting knowledge via finetuning results in brittle performance and poor generalization. We conduct a large-scale experimental study of finetuning the frontier Gemini v1.5 model family on a spectrum of datasets.
  arXiv  Detail & Related papers  (2025-03-07T20:35:31Z)
• Gradual Learning: Optimizing Fine-Tuning with Partially Mastered Knowledge in Large Language Models [51.20499954955646]
  Large language models (LLMs) acquire vast amounts of knowledge from extensive text corpora during the pretraining phase. In later stages such as fine-tuning and inference, the model may encounter knowledge not covered in the initial training. We propose a two-stage fine-tuning strategy to improve the model's overall test accuracy and knowledge retention.
  arXiv  Detail & Related papers  (2024-10-08T08:35:16Z)
• Does Fine-Tuning LLMs on New Knowledge Encourage Hallucinations? [33.702498916775426]
  We study the impact of new knowledge on the capability of the fine-tuned model to utilize its pre-existing knowledge. We demonstrate that large language models struggle to acquire new factual knowledge through fine-tuning. As the examples with new knowledge are eventually learned, they linearly increase the model's tendency to hallucinate.
  arXiv  Detail & Related papers  (2024-05-09T17:00:22Z)
• Unveiling Imitation Learning: Exploring the Impact of Data Falsity to Large Language Model [6.097530398802087]
  This paper explores the correlation between the degree of noise and its impact on language models through instruction tuning. Specifically, we found multiple intriguing findings of the correlation between the factuality of the dataset and instruction tuning.
  arXiv  Detail & Related papers  (2024-04-15T12:20:09Z)
• R-Tuning: Instructing Large Language Models to Say `I Don't Know' [66.11375475253007]
  Large language models (LLMs) have revolutionized numerous domains with their impressive performance but still face their challenges. Previous instruction tuning methods force the model to complete a sentence no matter whether the model knows the knowledge or not. We present a new approach called Refusal-Aware Instruction Tuning (R-Tuning) Experimental results demonstrate R-Tuning effectively improves a model's ability to answer known questions and refrain from answering unknown questions.
  arXiv  Detail & Related papers  (2023-11-16T08:45:44Z)
• Mitigating Temporal Misalignment by Discarding Outdated Facts [58.620269228776294]
  Large language models are often used under temporal misalignment, tasked with answering questions about the present. We propose fact duration prediction: the task of predicting how long a given fact will remain true. Our data and code are released publicly at https://github.com/mikejqzhang/mitigating_misalignment.
  arXiv  Detail & Related papers  (2023-05-24T07:30:08Z)
• Understanding Finetuning for Factual Knowledge Extraction from Language Models [1.7969777786551426]
  Language models (LMs) pretrained on large corpora of text from the web have been observed to contain large amounts of various types of knowledge about the world. We show that finetuning LMs on a set of factual knowledge makes them produce better answers to queries from a different set, thus making finetuned LMs a good candidate for knowledge extraction. We then consider two solutions to remedy the identified negative effect: 1- model mixing and 2- mixture finetuning with the LM's pre-training task.
  arXiv  Detail & Related papers  (2023-01-26T18:29:50Z)
• FactPEGASUS: Factuality-Aware Pre-training and Fine-tuning for Abstractive Summarization [91.46015013816083]
  We present FactPEG, an abstractive summarization model that addresses the problem of factuality during pre-training and fine-tuning. Our analysis suggests FactPEG is more factual than using the original pre-training objective in zero-shot and fewshot settings.
  arXiv  Detail & Related papers  (2022-05-16T17:39:14Z)
• BitFit: Simple Parameter-efficient Fine-tuning for Transformer-based Masked Language-models [51.53936551681613]
  We show that fine-tuning only the bias terms (or a subset of the bias terms) of pre-trained BERT models is competitive with (and sometimes better than) fine-tuning the entire model. They support the hypothesis that finetuning is mainly about exposing knowledge induced by language-modeling training, rather than learning new task-specific linguistic knowledge.
  arXiv  Detail & Related papers  (2021-06-18T16:09:21Z)
• On the Interplay Between Fine-tuning and Sentence-level Probing for Linguistic Knowledge in Pre-trained Transformers [24.858283637038422]
  We study three different pre-trained models: BERT, RoBERTa, and ALBERT. We find that for some probing tasks fine-tuning leads to substantial changes in accuracy. While fine-tuning indeed changes the representations of a pre-trained model, only in very few cases, fine-tuning has a positive effect on probing accuracy.
  arXiv  Detail & Related papers  (2020-10-06T10:54:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.