Entity-aware Transformers for Entity Search

• URL: http://arxiv.org/abs/2205.00820v1
• Date: Mon, 2 May 2022 11:53:59 GMT
• Title: Entity-aware Transformers for Entity Search
• Authors: Emma J. Gerritse, Faegheh Hasibi, Arjen P. de Vries
• Abstract summary: We show that the entity-enriched BERT model improves effectiveness on entity-oriented queries over a regular BERT model. We also show that the entity information provided by our entity-enriched model particularly helps queries related to less popular entities.
• Score: 6.107210856380526
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Pre-trained language models such as BERT have been a key ingredient to achieve state-of-the-art results on a variety of tasks in natural language processing and, more recently, also in information retrieval.Recent research even claims that BERT is able to capture factual knowledge about entity relations and properties, the information that is commonly obtained from knowledge graphs. This paper investigates the following question: Do BERT-based entity retrieval models benefit from additional entity information stored in knowledge graphs? To address this research question, we map entity embeddings into the same input space as a pre-trained BERT model and inject these entity embeddings into the BERT model. This entity-enriched language model is then employed on the entity retrieval task. We show that the entity-enriched BERT model improves effectiveness on entity-oriented queries over a regular BERT model, establishing a new state-of-the-art result for the entity retrieval task, with substantial improvements for complex natural language queries and queries requesting a list of entities with a certain property. Additionally, we show that the entity information provided by our entity-enriched model particularly helps queries related to less popular entities. Last, we observe empirically that the entity-enriched BERT models enable fine-tuning on limited training data, which otherwise would not be feasible due to the known instabilities of BERT in few-sample fine-tuning, thereby contributing to data-efficient training of BERT for entity search.

Related papers

• Entity Disambiguation via Fusion Entity Decoding [68.77265315142296]
  We propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions. We observe +1.5% improvements in end-to-end entity linking in the GERBIL benchmark compared with EntQA.
  arXiv  Detail & Related papers  (2024-04-02T04:27:54Z)
• Learning to Extract Structured Entities Using Language Models [52.281701191329]
  Recent advances in machine learning have significantly impacted the field of information extraction. We reformulate the task to be entity-centric, enabling the use of diverse metrics. We contribute to the field by introducing Structured Entity Extraction and proposing the Approximate Entity Set OverlaP metric.
  arXiv  Detail & Related papers  (2024-02-06T22:15:09Z)
• Memorization of Named Entities in Fine-tuned BERT Models [3.0177210416625115]
  We investigate the extent of named entity memorization in fine-tuned BERT models. We show that a fine-tuned BERT does not generate more named entities specific to the fine-tuning dataset than a BERT model that is pre-trained only.
  arXiv  Detail & Related papers  (2022-12-07T16:20:50Z)
• Representing Knowledge by Spans: A Knowledge-Enhanced Model for Information Extraction [7.077412533545456]
  We propose a new pre-trained model that learns representations of both entities and relationships simultaneously. By encoding spans efficiently with span modules, our model can represent both entities and their relationships but requires fewer parameters than existing models.
  arXiv  Detail & Related papers  (2022-08-20T07:32:25Z)
• A Simple but Effective Pluggable Entity Lookup Table for Pre-trained Language Models [93.39977756450354]
  We propose to build a simple but effective Pluggable Entity Lookup Table (PELT) on demand. PELT can be compatibly plugged as inputs to infuse entity supplemental knowledge into pre-trained language models. Experiments on knowledge-related tasks demonstrate that our method, PELT, can flexibly and effectively transfer entity knowledge from related corpora into PLMs.
  arXiv  Detail & Related papers  (2022-02-27T16:30:22Z)
• Fast and Effective Biomedical Entity Linking Using a Dual Encoder [48.86736921025866]
  We propose a BERT-based dual encoder model that resolves multiple mentions in a document in one shot. We show that our proposed model is multiple times faster than existing BERT-based models while being competitive in accuracy for biomedical entity linking.
  arXiv  Detail & Related papers  (2021-03-08T19:32:28Z)
• HittER: Hierarchical Transformers for Knowledge Graph Embeddings [85.93509934018499]
  We propose Hitt to learn representations of entities and relations in a complex knowledge graph. Experimental results show that Hitt achieves new state-of-the-art results on multiple link prediction. We additionally propose a simple approach to integrate Hitt into BERT and demonstrate its effectiveness on two Freebase factoid answering datasets.
  arXiv  Detail & Related papers  (2020-08-28T18:58:15Z)
• Table Search Using a Deep Contextualized Language Model [20.041167804194707]
  In this paper, we use the deep contextualized language model BERT for the task of ad hoc table retrieval. We propose an approach that incorporates features from prior literature on table retrieval and jointly trains them with BERT.
  arXiv  Detail & Related papers  (2020-05-19T04:18:04Z)
• Interpretable Entity Representations through Large-Scale Typing [61.4277527871572]
  We present an approach to creating entity representations that are human readable and achieve high performance out of the box. Our representations are vectors whose values correspond to posterior probabilities over fine-grained entity types. We show that it is possible to reduce the size of our type set in a learning-based way for particular domains.
  arXiv  Detail & Related papers  (2020-04-30T23:58:03Z)
• Investigating Entity Knowledge in BERT with Simple Neural End-To-End Entity Linking [8.265860641797996]
  We propose an extreme simplification of the entity linking setup that works surprisingly well. We show that this model improves the entity representations over plain BERT. We also investigate the usefulness of entity-aware token-representations in the text-understanding benchmark GLUE.
  arXiv  Detail & Related papers  (2020-03-11T18:23:00Z)
• Improving Entity Linking by Modeling Latent Entity Type Information [25.33342677359822]
  We propose to inject latent entity type information into the entity embeddings based on pre-trained BERT. In addition, we integrate a BERT-based entity similarity score into the local context model of a state-of-the-art model to better capture latent entity type information. Our model significantly outperforms the state-of-the-art entity linking models on standard benchmark (AIDA-CoNLL)
  arXiv  Detail & Related papers  (2020-01-06T09:18:29Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.