OpenPI2.0: An Improved Dataset for Entity Tracking in Texts

• URL: http://arxiv.org/abs/2305.14603v2
• Date: Thu, 25 Jan 2024 18:15:31 GMT
• Title: OpenPI2.0: An Improved Dataset for Entity Tracking in Texts
• Authors: Li Zhang, Hainiu Xu, Abhinav Kommula, Chris Callison-Burch, Niket Tandon
• Abstract summary: An earlier dataset, OpenPI, provided crowdsourced annotations of entity state changes in text. We present an improved dataset, OpenPI2.0, where entities and attributes are fully canonicalized and additional entity salience annotations are added. We show that using state changes of salient entities as a chain-of-thought prompt, downstream performance is improved on tasks such as question answering and classical planning.
• Score: 36.84433853139042
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Much text describes a changing world (e.g., procedures, stories, newswires), and understanding them requires tracking how entities change. An earlier dataset, OpenPI, provided crowdsourced annotations of entity state changes in text. However, a major limitation was that those annotations were free-form and did not identify salient changes, hampering model evaluation. To overcome these limitations, we present an improved dataset, OpenPI2.0, where entities and attributes are fully canonicalized and additional entity salience annotations are added. On our fairer evaluation setting, we find that current state-of-the-art language models are far from competent. We also show that using state changes of salient entities as a chain-of-thought prompt, downstream performance is improved on tasks such as question answering and classical planning, outperforming the setting involving all related entities indiscriminately. We offer OpenPI2.0 for the continued development of models that can understand the dynamics of entities in text.

Related papers

• Dynamic Retrieval-Augmented Generation [4.741884506444161]
  We propose a novel approach for the Dynamic Retrieval-Augmented Generation (DRAG) DRAG injects compressed embeddings of the retrieved entities into the generative model. Our approach achieves several targets: (1) lifting the length limitations of the context window, saving on the prompt size; (2) allowing huge expansion of the number of retrieval entities available for the context; (3) alleviating the problem of misspelling or failing to find relevant entity names.
  arXiv  Detail & Related papers  (2023-12-14T14:26:57Z)
• OpenPI-C: A Better Benchmark and Stronger Baseline for Open-Vocabulary State Tracking [55.62705574507595]
  OpenPI is the only dataset annotated for open-vocabulary state tracking. We categorize 3 types of problems on the procedure level, step level and state change level respectively. For the evaluation metric, we propose a cluster-based metric to fix the original metric's preference for repetition.
  arXiv  Detail & Related papers  (2023-06-01T16:48:20Z)
• Understand the Dynamic World: An End-to-End Knowledge Informed Framework for Open Domain Entity State Tracking [15.421012879083463]
  Open domain entity state tracking aims to predict reasonable state changes of entities (i.e., [attribute] of [entity] was [before_state] and [after_state] afterwards) given the action descriptions. It's challenging as the model needs to predict an arbitrary number of entity state changes caused by the action while most of the entities are implicitly relevant to the actions and their attributes as well as states are from open vocabularies. We propose a novel end-to-end Knowledge Informed framework for open domain Entity State Tracking, namely KIEST, which explicitly retrieves the relevant entities and attributes from
  arXiv  Detail & Related papers  (2023-04-26T22:45:30Z)
• Modeling Entities as Semantic Points for Visual Information Extraction in the Wild [55.91783742370978]
  We propose an alternative approach to precisely and robustly extract key information from document images. We explicitly model entities as semantic points, i.e., center points of entities are enriched with semantic information describing the attributes and relationships of different entities. The proposed method can achieve significantly enhanced performance on entity labeling and linking, compared with previous state-of-the-art models.
  arXiv  Detail & Related papers  (2023-03-23T08:21:16Z)
• Coalescing Global and Local Information for Procedural Text Understanding [70.10291759879887]
  A complete procedural understanding solution should combine three core aspects: local and global views of the inputs, and global view of outputs. In this paper, we propose Coalescing Global and Local InformationCG, a new model that builds entity and time representations. Experiments on a popular procedural text understanding dataset show that our model achieves state-of-the-art results.
  arXiv  Detail & Related papers  (2022-08-26T19:16:32Z)
• Entity Cloze By Date: What LMs Know About Unseen Entities [79.34707800653597]
  Language models (LMs) are typically trained once on a large-scale corpus and used for years without being updated. We propose a framework to analyze what LMs can infer about new entities that did not exist when the LMs were pretrained. We derive a dataset of entities indexed by their origination date and paired with their English Wikipedia articles, from which we can find sentences about each entity.
  arXiv  Detail & Related papers  (2022-05-05T17:59:31Z)
• Towards Entity Alignment in the Open World: An Unsupervised Approach [29.337157862514204]
  It is a pivotal step for integrating knowledge graphs (KGs) to increase knowledge coverage and quality. State-of-the-art solutions tend to rely on labeled data for model training. We offer an unsupervised framework that performs entity alignment in the open world.
  arXiv  Detail & Related papers  (2021-01-26T03:10:24Z)
• LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention [37.111204321059084]
  We propose new pretrained contextualized representations of words and entities based on the bidirectional transformer. Our model is trained using a new pretraining task based on the masked language model of BERT. We also propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer.
  arXiv  Detail & Related papers  (2020-10-02T15:38:03Z)
• Interpretable Entity Representations through Large-Scale Typing [61.4277527871572]
  We present an approach to creating entity representations that are human readable and achieve high performance out of the box. Our representations are vectors whose values correspond to posterior probabilities over fine-grained entity types. We show that it is possible to reduce the size of our type set in a learning-based way for particular domains.
  arXiv  Detail & Related papers  (2020-04-30T23:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.