DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entities

• URL: http://arxiv.org/abs/2410.07722v2
• Date: Tue, 15 Oct 2024 19:58:37 GMT
• Title: DyVo: Dynamic Vocabularies for Learned Sparse Retrieval with Entities
• Authors: Thong Nguyen, Shubham Chatterjee, Sean MacAvaney, Iain Mackie, Jeff Dalton, Andrew Yates,
• Abstract summary: We enhance the Learned Sparse Retrieval (LSR) model with Wikipedia concepts and entities. In experiments across three entity-rich document ranking datasets, the resulting DyVo model substantially outperforms state-of-the-art baselines.
• Score: 29.716152560414738
• License:
• Abstract: Learned Sparse Retrieval (LSR) models use vocabularies from pre-trained transformers, which often split entities into nonsensical fragments. Splitting entities can reduce retrieval accuracy and limits the model's ability to incorporate up-to-date world knowledge not included in the training data. In this work, we enhance the LSR vocabulary with Wikipedia concepts and entities, enabling the model to resolve ambiguities more effectively and stay current with evolving knowledge. Central to our approach is a Dynamic Vocabulary (DyVo) head, which leverages existing entity embeddings and an entity retrieval component that identifies entities relevant to a query or document. We use the DyVo head to generate entity weights, which are then merged with word piece weights to create joint representations for efficient indexing and retrieval using an inverted index. In experiments across three entity-rich document ranking datasets, the resulting DyVo model substantially outperforms state-of-the-art baselines.

Related papers

• Web-Scale Visual Entity Recognition: An LLM-Driven Data Approach [56.55633052479446]
  Web-scale visual entity recognition presents significant challenges due to the lack of clean, large-scale training data. We propose a novel methodology to curate such a dataset, leveraging a multimodal large language model (LLM) for label verification, metadata generation, and rationale explanation. Experiments demonstrate that models trained on this automatically curated data achieve state-of-the-art performance on web-scale visual entity recognition tasks.
  arXiv  Detail & Related papers  (2024-10-31T06:55:24Z)
• Retrieval-Enhanced Named Entity Recognition [1.2187048691454239]
  RENER is a technique for named entity recognition using autoregressive language models based on In-Context Learning and information retrieval techniques. Experimental results show that in the CrossNER collection we achieve state-of-the-art performance with the proposed technique.
  arXiv  Detail & Related papers  (2024-10-17T01:12:48Z)
• DIM: Dynamic Integration of Multimodal Entity Linking with Large Language Model [16.20833396645551]
  We propose dynamic entity extraction using ChatGPT, which dynamically extracts entities and enhances datasets. We also propose a method: Dynamically Integrate Multimodal information with knowledge base (DIM), employing the capability of the Large Language Model (LLM) for visual understanding.
  arXiv  Detail & Related papers  (2024-06-27T15:18:23Z)
• Entity Disambiguation via Fusion Entity Decoding [68.77265315142296]
  We propose an encoder-decoder model to disambiguate entities with more detailed entity descriptions. We observe +1.5% improvements in end-to-end entity linking in the GERBIL benchmark compared with EntQA.
  arXiv  Detail & Related papers  (2024-04-02T04:27:54Z)
• Language Models As Semantic Indexers [78.83425357657026]
  We introduce LMIndexer, a self-supervised framework to learn semantic IDs with a generative language model. We show the high quality of the learned IDs and demonstrate their effectiveness on three tasks including recommendation, product search, and document retrieval.
  arXiv  Detail & Related papers  (2023-10-11T18:56:15Z)
• UnifieR: A Unified Retriever for Large-Scale Retrieval [84.61239936314597]
  Large-scale retrieval is to recall relevant documents from a huge collection given a query. Recent retrieval methods based on pre-trained language models (PLM) can be coarsely categorized into either dense-vector or lexicon-based paradigms. We propose a new learning framework, UnifieR which unifies dense-vector and lexicon-based retrieval in one model with a dual-representing capability.
  arXiv  Detail & Related papers  (2022-05-23T11:01:59Z)
• Improving Entity Linking through Semantic Reinforced Entity Embeddings [16.868791358905916]
  We propose a method to inject fine-grained semantic information into entity embeddings to reduce the distinctiveness and facilitate the learning of contextual commonality. Based on our entity embeddings, we achieved new sate-of-the-art performance on entity linking.
  arXiv  Detail & Related papers  (2021-06-16T00:27:56Z)
• Interpretable and Low-Resource Entity Matching via Decoupling Feature Learning from Decision Making [22.755892575582788]
  Entity Matching aims at recognizing entity records that denote the same real-world object. We propose a novel EM framework that consists of Heterogeneous Information Fusion (HIF) and Key Attribute Tree (KAT) Induction. Our method is highly efficient and outperforms SOTA EM models in most cases.
  arXiv  Detail & Related papers  (2021-06-08T08:27:31Z)
• LUKE: Deep Contextualized Entity Representations with Entity-aware Self-attention [37.111204321059084]
  We propose new pretrained contextualized representations of words and entities based on the bidirectional transformer. Our model is trained using a new pretraining task based on the masked language model of BERT. We also propose an entity-aware self-attention mechanism that is an extension of the self-attention mechanism of the transformer.
  arXiv  Detail & Related papers  (2020-10-02T15:38:03Z)
• Autoregressive Entity Retrieval [55.38027440347138]
  Entities are at the center of how we represent and aggregate knowledge. The ability to retrieve such entities given a query is fundamental for knowledge-intensive tasks such as entity linking and open-domain question answering. We propose GENRE, the first system that retrieves entities by generating their unique names, left to right, token-by-token in an autoregressive fashion.
  arXiv  Detail & Related papers  (2020-10-02T10:13:31Z)
• Interpretable Entity Representations through Large-Scale Typing [61.4277527871572]
  We present an approach to creating entity representations that are human readable and achieve high performance out of the box. Our representations are vectors whose values correspond to posterior probabilities over fine-grained entity types. We show that it is possible to reduce the size of our type set in a learning-based way for particular domains.
  arXiv  Detail & Related papers  (2020-04-30T23:58:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.