Physics of Language Models: Part 3.1, Knowledge Storage and Extraction

• URL: http://arxiv.org/abs/2309.14316v3
• Date: Tue, 16 Jul 2024 10:22:51 GMT
• Title: Physics of Language Models: Part 3.1, Knowledge Storage and Extraction
• Authors: Zeyuan Allen-Zhu, Yuanzhi Li,
• Abstract summary: Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data.
• Score: 51.68385617116854
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) can store a vast amount of world knowledge, often extractable via question-answering (e.g., "What is Abraham Lincoln's birthday?"). However, do they answer such questions based on exposure to similar questions during training (i.e., cheating), or by genuinely learning to extract knowledge from sources like Wikipedia? In this paper, we investigate this issue using a controlled biography dataset. We find a strong correlation between the model's ability to extract knowledge and various diversity measures of the training data. $\textbf{Essentially}$, for knowledge to be reliably extracted, it must be sufficiently augmented (e.g., through paraphrasing, sentence shuffling, translations) $\textit{during pretraining}$. Without such augmentation, knowledge may be memorized but not extractable, leading to 0% accuracy, regardless of subsequent instruction fine-tuning. To understand why this occurs, we employ (nearly) linear probing to demonstrate a strong connection between the observed correlation and how the model internally encodes knowledge -- whether it is linearly encoded in the hidden embeddings of entity names or distributed across other token embeddings in the training text. This paper provides $\textbf{several key recommendations for LLM pretraining in the industry}$: (1) rewrite the pretraining data -- using small, auxiliary models -- to provide knowledge augmentation, and (2) incorporate more instruction-finetuning data into the pretraining stage before it becomes too late.

Related papers

• Enhancing Contextual Understanding in Large Language Models through Contrastive Decoding [9.2433070542025]
  Large language models (LLMs) tend to inadequately integrate input context during text generation. We introduce a novel approach integrating contrastive decoding with adversarial irrelevant passages as negative samples.
  arXiv  Detail & Related papers  (2024-05-04T20:38:41Z)
• Physics of Language Models: Part 3.3, Knowledge Capacity Scaling Laws [51.68385617116854]
  Scaling laws describe the relationship between the size of language models and their capabilities. We focus on factual knowledge represented as domains, such as (USA, capital, Washington D.C.) from a Wikipedia page. A 7B model can store 14B bits of knowledge, surpassing the English Wikipedia and textbooks combined.
  arXiv  Detail & Related papers  (2024-04-08T11:11:31Z)
• Source-Aware Training Enables Knowledge Attribution in Language Models [81.13048060332775]
  Intrinsic source citation can enhance transparency, interpretability, and verifiability. Our training recipe can enable faithful attribution to the pretraining data without a substantial impact on the model's perplexity.
  arXiv  Detail & Related papers  (2024-04-01T09:39:38Z)
• Robust and Scalable Model Editing for Large Language Models [75.95623066605259]
  We propose EREN (Edit models by REading Notes) to improve the scalability and robustness of LLM editing. Unlike existing techniques, it can integrate knowledge from multiple edits, and correctly respond to syntactically similar but semantically unrelated inputs.
  arXiv  Detail & Related papers  (2024-03-26T06:57:23Z)
• Decouple knowledge from parameters for plug-and-play language modeling [77.5601135412186]
  We introduce PlugLM, a pre-training model with differentiable plug-in memory(DPM) The key intuition is to decouple the knowledge storage from model parameters with an editable and scalable key-value memory. PlugLM obtains 3.95 F1 improvements across four domains on average without any in-domain pre-training.
  arXiv  Detail & Related papers  (2023-05-19T10:01:55Z)
• RECKONING: Reasoning through Dynamic Knowledge Encoding [51.076603338764706]
  We show that language models can answer questions by reasoning over knowledge provided as part of the context. In these situations, the model fails to distinguish the knowledge that is necessary to answer the question. We propose teaching the model to reason more robustly by folding the provided contextual knowledge into the model's parameters.
  arXiv  Detail & Related papers  (2023-05-10T17:54:51Z)
• Zero-shot Commonsense Question Answering with Cloze Translation and Consistency Optimization [20.14487209460865]
  We investigate four translation methods that can translate natural questions into cloze-style sentences. We show that our methods are complementary datasets to a knowledge base improved model, and combining them can lead to state-of-the-art zero-shot performance.
  arXiv  Detail & Related papers  (2022-01-01T07:12:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.