Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models

• URL: http://arxiv.org/abs/2310.16570v2
• Date: Mon, 4 Dec 2023 19:23:33 GMT
• Title: Give Me the Facts! A Survey on Factual Knowledge Probing in Pre-trained Language Models
• Authors: Paul Youssef, Osman Alperen Kora\c{s}, Meijie Li, J\"org Schl\"otterer, Christin Seifert
• Abstract summary: Pre-trained Language Models (PLMs) are trained on vast unlabeled data, rich in world knowledge. This has sparked the interest of the community in quantifying the amount of factual knowledge present in PLMs. In this work, we survey methods and datasets that are used to probe PLMs for factual knowledge.
• Score: 2.3981254787726067
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: Pre-trained Language Models (PLMs) are trained on vast unlabeled data, rich in world knowledge. This fact has sparked the interest of the community in quantifying the amount of factual knowledge present in PLMs, as this explains their performance on downstream tasks, and potentially justifies their use as knowledge bases. In this work, we survey methods and datasets that are used to probe PLMs for factual knowledge. Our contributions are: (1) We propose a categorization scheme for factual probing methods that is based on how their inputs, outputs and the probed PLMs are adapted; (2) We provide an overview of the datasets used for factual probing; (3) We synthesize insights about knowledge retention and prompt optimization in PLMs, analyze obstacles to adopting PLMs as knowledge bases and outline directions for future work.

Related papers

• KBAlign: Efficient Self Adaptation on Specific Knowledge Bases [75.78948575957081]
  Large language models (LLMs) usually rely on retrieval-augmented generation to exploit knowledge materials in an instant manner. We propose KBAlign, an approach designed for efficient adaptation to downstream tasks involving knowledge bases. Our method utilizes iterative training with self-annotated data such as Q&A pairs and revision suggestions, enabling the model to grasp the knowledge content efficiently.
  arXiv  Detail & Related papers  (2024-11-22T08:21:03Z)
• Enhancing Fact Retrieval in PLMs through Truthfulness [2.722191152967056]
  Pre-trained Language Models (PLMs) encode various facts about the world at their pre-training phase as they are trained to predict the next or missing word in a sentence. Recent work shows that the hidden states of PLMs can be leveraged to determine the truthfulness of the PLMs' inputs. In this work, we investigate the use of a helper model to improve fact retrieval.
  arXiv  Detail & Related papers  (2024-10-17T14:00:13Z)
• The Queen of England is not England's Queen: On the Lack of Factual Coherency in PLMs [2.9443699603751536]
  Factual knowledge encoded in Pre-trained Language Models (PLMs) enriches their representations and justifies their use as knowledge bases. Previous work has focused on probing PLMs for factual knowledge by measuring how often they can correctly predict an object entity given a subject and a relation. In this work, we consider a complementary aspect, namely the coherency of factual knowledge in PLMs, i.e., how often can PLMs predict the subject entity given its initial prediction of the object entity.
  arXiv  Detail & Related papers  (2024-02-02T14:42:09Z)
• Eva-KELLM: A New Benchmark for Evaluating Knowledge Editing of LLMs [54.22416829200613]
  Eva-KELLM is a new benchmark for evaluating knowledge editing of large language models. Experimental results indicate that the current methods for knowledge editing using raw documents are not effective in yielding satisfactory results.
  arXiv  Detail & Related papers  (2023-08-19T09:17:19Z)
• Measuring and Modifying Factual Knowledge in Large Language Models [2.8427946758947304]
  Large Language Models store an extensive amount of factual knowledge obtained from vast collections of text. We employ information theory-based measurements to provide a framework estimating the factual knowledge contained within large language models.
  arXiv  Detail & Related papers  (2023-06-09T21:25:48Z)
• Measuring the Knowledge Acquisition-Utilization Gap in Pretrained Language Models [26.342351417963965]
  Pre-trained language models (PLMs) have shown evidence of acquiring vast amounts of knowledge. It remains unclear how much of this parametric knowledge is actually usable in performing downstream tasks. We propose a systematic framework to measure parametric knowledge utilization in PLMs.
  arXiv  Detail & Related papers  (2023-05-24T06:26:11Z)
• Knowledge Rumination for Pre-trained Language Models [77.55888291165462]
  We propose a new paradigm dubbed Knowledge Rumination to help the pre-trained language model utilize related latent knowledge without retrieving it from the external corpus. We apply the proposed knowledge rumination to various language models, including RoBERTa, DeBERTa, and GPT-3.
  arXiv  Detail & Related papers  (2023-05-15T15:47:09Z)
• A Survey of Knowledge Enhanced Pre-trained Language Models [78.56931125512295]
  We present a comprehensive review of Knowledge Enhanced Pre-trained Language Models (KE-PLMs) For NLU, we divide the types of knowledge into four categories: linguistic knowledge, text knowledge, knowledge graph (KG) and rule knowledge. The KE-PLMs for NLG are categorized into KG-based and retrieval-based methods.
  arXiv  Detail & Related papers  (2022-11-11T04:29:02Z)
• ElitePLM: An Empirical Study on General Language Ability Evaluation of Pretrained Language Models [78.08792285698853]
  We present a large-scale empirical study on general language ability evaluation of pretrained language models (ElitePLM) Our empirical results demonstrate that: (1) PLMs with varying training objectives and strategies are good at different ability tests; (2) fine-tuning PLMs in downstream tasks is usually sensitive to the data size and distribution; and (3) PLMs have excellent transferability between similar tasks.
  arXiv  Detail & Related papers  (2022-05-03T14:18:10Z)
• Knowledgeable Salient Span Mask for Enhancing Language Models as Knowledge Base [51.55027623439027]
  We develop two solutions to help the model learn more knowledge from unstructured text in a fully self-supervised manner. To our best knowledge, we are the first to explore fully self-supervised learning of knowledge in continual pre-training.
  arXiv  Detail & Related papers  (2022-04-17T12:33:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.