Source-Aware Training Enables Knowledge Attribution in Language Models

• URL: http://arxiv.org/abs/2404.01019v3
• Date: Tue, 13 Aug 2024 03:55:35 GMT
• Title: Source-Aware Training Enables Knowledge Attribution in Language Models
• Authors: Muhammad Khalifa, David Wadden, Emma Strubell, Honglak Lee, Lu Wang, Iz Beltagy, Hao Peng,
• Abstract summary: Intrinsic source citation can enhance transparency, interpretability, and verifiability. Our training recipe can enable faithful attribution to the pretraining data without a substantial impact on the model's perplexity.
• Score: 81.13048060332775
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models (LLMs) learn a vast amount of knowledge during pretraining, but they are often oblivious to the source(s) of such knowledge. We investigate the problem of intrinsic source citation, where LLMs are required to cite the pretraining source supporting a generated response. Intrinsic source citation can enhance LLM transparency, interpretability, and verifiability. To give LLMs such ability, we explore source-aware training -- a recipe that involves (i) training the LLM to associate unique source document identifiers with the knowledge in each document, followed by (ii) an instruction-tuning stage to teach the LLM to cite a supporting pretraining source when prompted. Source-aware training borrows from existing pretraining/fine-tuning frameworks and requires minimal changes to the model architecture or implementation. Through experiments on synthetic data, we demonstrate that our training recipe can enable faithful attribution to the pretraining data without a substantial impact on the model's perplexity compared to standard pretraining. Our findings also highlight the importance of pretraining data augmentation in achieving attribution. Code and data available here: \url{https://github.com/mukhal/intrinsic-source-citation}

Related papers

• Extracting Memorized Training Data via Decomposition [24.198975804570072]
  We demonstrate a simple, query-based decompositional method to extract news articles from two frontier Large Language Models. We extract at least one sentence from 73 articles, and over 20% of verbatim sentences from 6 articles. If replicable at scale, this training data extraction methodology could expose new LLM security and safety vulnerabilities.
  arXiv  Detail & Related papers  (2024-09-18T23:59:32Z)
• How Do Large Language Models Acquire Factual Knowledge During Pretraining? [36.59608982935844]
  We study how large language models (LLMs) acquire factual knowledge during pretraining. Findings reveal several important insights into the dynamics of factual knowledge acquisition during pretraining.
  arXiv  Detail & Related papers  (2024-06-17T17:54:40Z)
• SPOT: Text Source Prediction from Originality Score Thresholding [6.790905400046194]
  countermeasures aim at detecting misinformation, usually involve domain specific models trained to recognize the relevance of any information. Instead of evaluating the validity of the information, we propose to investigate LLM generated text from the perspective of trust.
  arXiv  Detail & Related papers  (2024-05-30T21:51:01Z)
• Alpaca against Vicuna: Using LLMs to Uncover Memorization of LLMs [61.04246774006429]
  We introduce a black-box prompt optimization method that uses an attacker LLM agent to uncover higher levels of memorization in a victim agent. We observe that our instruction-based prompts generate outputs with 23.7% higher overlap with training data compared to the baseline prefix-suffix measurements. Our findings show that instruction-tuned models can expose pre-training data as much as their base-models, if not more so, and using instructions proposed by other LLMs can open a new avenue of automated attacks.
  arXiv  Detail & Related papers  (2024-03-05T19:32:01Z)
• VILA: On Pre-training for Visual Language Models [74.08039416548209]
  We study the design options for VLM pre-training through step-by-step controllable comparisons. We build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-12T18:58:18Z)
• Source Attribution for Large Language Model-Generated Data [57.85840382230037]
  It is imperative to be able to perform source attribution by identifying the data provider who contributed to the generation of a synthetic text. We show that this problem can be tackled by watermarking. We propose a source attribution framework that satisfies these key properties due to our algorithmic designs.
  arXiv  Detail & Related papers  (2023-10-01T12:02:57Z)
• Physics of Language Models: Part 3.1, Knowledge Storage and Extraction [51.68385617116854]
  Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data.
  arXiv  Detail & Related papers  (2023-09-25T17:37:20Z)
• CodeGen2: Lessons for Training LLMs on Programming and Natural Languages [116.74407069443895]
  We unify encoder and decoder-based models into a single prefix-LM. For learning methods, we explore the claim of a "free lunch" hypothesis. For data distributions, the effect of a mixture distribution and multi-epoch training of programming and natural languages on model performance is explored.
  arXiv  Detail & Related papers  (2023-05-03T17:55:25Z)
• REALM: Retrieval-Augmented Language Model Pre-Training [37.3178586179607]
  We augment language model pre-training with a latent knowledge retriever, which allows the model to retrieve and attend over documents from a large corpus such as Wikipedia. For the first time, we show how to pre-train such a knowledge retriever in an unsupervised manner. We demonstrate the effectiveness of Retrieval-Augmented Language Model pre-training (REALM) by fine-tuning on the challenging task of Open-domain Question Answering (Open-QA)
  arXiv  Detail & Related papers  (2020-02-10T18:40:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.