Entity Cloze By Date: What LMs Know About Unseen Entities

• URL: http://arxiv.org/abs/2205.02832v1
• Date: Thu, 5 May 2022 17:59:31 GMT
• Title: Entity Cloze By Date: What LMs Know About Unseen Entities
• Authors: Yasumasa Onoe, Michael J.Q. Zhang, Eunsol Choi, Greg Durrett
• Abstract summary: Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity.
• Score: 79.34707800653597
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. However, in a dynamic world, new entities constantly arise. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity. We evaluate LMs' perplexity on masked spans within these sentences. We show that models more informed about the entities, such as those with access to a textual definition of them, achieve lower perplexity on this benchmark. Our experimental results demonstrate that making inferences about new entities remains difficult for LMs. Given its wide coverage on entity knowledge and temporal indexing, our dataset can be used to evaluate LMs and techniques designed to modify or extend their knowledge. Our automatic data collection pipeline can be easily used to continually update our benchmark.

Related papers

• Learning from Natural Language Explanations for Generalizable Entity Matching [19.978468744557173]
  We re-cast entity matching as a conditional generation task as opposed to binary classification. This enables us to "distill" LLM reasoning into smaller entity matching models via natural language explanations.
  arXiv  Detail & Related papers  (2024-06-13T17:08:58Z)
• Reliable, Adaptable, and Attributable Language Models with Retrieval [144.26890121729514]
  Parametric language models (LMs) are trained on vast amounts of web data. They face practical challenges such as hallucinations, difficulty in adapting to new data distributions, and a lack of verifiability. We advocate for retrieval-augmented LMs to replace parametric LMs as the next generation of LMs.
  arXiv  Detail & Related papers  (2024-03-05T18:22:33Z)
• Can LMs Learn New Entities from Descriptions? Challenges in Propagating Injected Knowledge [72.63368052592004]
  We study LMs' abilities to make inferences based on injected facts (or propagate those facts) We find that existing methods for updating knowledge show little propagation of injected knowledge. Yet, prepending entity definitions in an LM's context improves performance across all settings.
  arXiv  Detail & Related papers  (2023-05-02T17:59:46Z)
• Augmented Language Models: a Survey [55.965967655575454]
  This survey reviews works in which language models (LMs) are augmented with reasoning skills and the ability to use tools. We refer to them as Augmented Language Models (ALMs) The missing token objective allows ALMs to learn to reason, use tools, and even act, while still performing standard natural language tasks.
  arXiv  Detail & Related papers  (2023-02-15T18:25:52Z)
• TempEL: Linking Dynamically Evolving and Newly Emerging Entities [50.980331847622026]
  In our continuously evolving world, entities change over time and new, previously non-existing or unknown, entities appear. We study how this evolutionary scenario impacts the performance on a well established entity linking (EL) task. We introduce TempEL, an entity linking dataset that consists of time-stratified English Wikipedia snapshots from 2013 to 2022.
  arXiv  Detail & Related papers  (2023-02-05T22:34:36Z)
• IELM: An Open Information Extraction Benchmark for Pre-Trained Language Models [75.48081086368606]
  We introduce a new open information extraction (OIE) benchmark for pre-trained language models (LM) We create an OIE benchmark aiming to fully examine the open relational information present in the pre-trained LMs. Surprisingly, pre-trained LMs are able to obtain competitive performance on both standard OIE datasets.
  arXiv  Detail & Related papers  (2022-10-25T16:25:00Z)
• TemporalWiki: A Lifelong Benchmark for Training and Evaluating Ever-Evolving Language Models [31.900232508466928]
  TemporalWiki is a lifelong benchmark for ever-evolving Language Models (LMs) It allows researchers to periodically track an LM's ability to retain previous knowledge and acquire updated/new knowledge at each point in time. We find that training an LM on the diff data through continual learning methods achieves similar or better perplexity than on the entire snapshot in our benchmark with 12 times less computational cost.
  arXiv  Detail & Related papers  (2022-04-29T16:40:07Z)
• Metadata Shaping: Natural Language Annotations for the Tail [4.665656172490747]
  Language models (LMs) have made remarkable progress, but still struggle to generalize beyond the training data to rare linguistic patterns. We propose metadata shaping, a method in which readily available metadata, such as entity descriptions and categorical tags, are appended to examples based on information theoretic metrics. With no changes to the LM whatsoever, metadata shaping exceeds the BERT-baseline by up to 5.3 F1 points, and achieves or competes with state-of-the-art results.
  arXiv  Detail & Related papers  (2021-10-16T01:00:47Z)
• Towards Continual Entity Learning in Language Models for Conversational Agents [0.5330240017302621]
  We introduce entity-aware language models (EALM), where we integrate entity models trained on catalogues of entities into pre-trained LMs. Our combined language model adaptively adds information from the entity models into the pre-trained LM depending on the sentence context. We show significant perplexity improvements on task-oriented dialogue datasets, especially on long-tailed utterances.
  arXiv  Detail & Related papers  (2021-07-30T21:10:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.