Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction

• URL: http://arxiv.org/abs/2203.13088v1
• Date: Thu, 24 Mar 2022 14:28:07 GMT
• Title: Introducing Neural Bag of Whole-Words with ColBERTer: Contextualized Late Interactions using Enhanced Reduction
• Authors: Sebastian Hofst\"atter, Omar Khattab, Sophia Althammer, Mete Sertkan, Allan Hanbury
• Abstract summary: ColBERTer is a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. For its multi-vector component, ColBERTer reduces the number of stored per document by learning unique whole-word representations for the terms in each document. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5x, while maintaining effectiveness.
• Score: 10.749746283569847
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent progress in neural information retrieval has demonstrated large gains in effectiveness, while often sacrificing the efficiency and interpretability of the neural model compared to classical approaches. This paper proposes ColBERTer, a neural retrieval model using contextualized late interaction (ColBERT) with enhanced reduction. Along the effectiveness Pareto frontier, ColBERTer's reductions dramatically lower ColBERT's storage requirements while simultaneously improving the interpretability of its token-matching scores. To this end, ColBERTer fuses single-vector retrieval, multi-vector refinement, and optional lexical matching components into one model. For its multi-vector component, ColBERTer reduces the number of stored vectors per document by learning unique whole-word representations for the terms in each document and learning to identify and remove word representations that are not essential to effective scoring. We employ an explicit multi-task, multi-stage training to facilitate using very small vector dimensions. Results on the MS MARCO and TREC-DL collection show that ColBERTer can reduce the storage footprint by up to 2.5x, while maintaining effectiveness. With just one dimension per token in its smallest setting, ColBERTer achieves index storage parity with the plaintext size, with very strong effectiveness results. Finally, we demonstrate ColBERTer's robustness on seven high-quality out-of-domain collections, yielding statistically significant gains over traditional retrieval baselines.

Related papers

• Jina-ColBERT-v2: A General-Purpose Multilingual Late Interaction Retriever [6.221757399678299]
  ColBERT's late interaction scoring approximates the joint query-document attention seen in cross-encoders. Our new model, Jina-ColBERT-v2, demonstrates strong performance across a range of English and multilingual retrieval tasks.
  arXiv  Detail & Related papers  (2024-08-29T16:21:00Z)
• SPLATE: Sparse Late Interaction Retrieval [13.607085390630647]
  SPLATE is a lightweight adaptation of the ColBERTv2 model which learns an MLM adapter'' Our pipeline achieves the same effectiveness as the PLAID ColBERTv2 engine by re-ranking 50 documents that can be retrieved under 10ms.
  arXiv  Detail & Related papers  (2024-04-22T07:51:13Z)
• Beyond Two-Tower Matching: Learning Sparse Retrievable Cross-Interactions for Recommendation [80.19762472699814]
  Two-tower models are a prevalent matching framework for recommendation, which have been widely deployed in industrial applications. It suffers two main challenges, including limited feature interaction capability and reduced accuracy in online serving. We propose a new matching paradigm named SparCode, which supports not only sophisticated feature interactions but also efficient retrieval.
  arXiv  Detail & Related papers  (2023-11-30T03:13:36Z)
• Improving Out-of-Distribution Generalization of Neural Rerankers with Contextualized Late Interaction [52.63663547523033]
  Late interaction, the simplest form of multi-vector, is also helpful to neural rerankers that only use the [] vector to compute the similarity score. We show that the finding is consistent across different model sizes and first-stage retrievers of diverse natures.
  arXiv  Detail & Related papers  (2023-02-13T18:42:17Z)
• Enriching Relation Extraction with OpenIE [70.52564277675056]
  Relation extraction (RE) is a sub-discipline of information extraction (IE) In this work, we explore how recent approaches for open information extraction (OpenIE) may help to improve the task of RE. Our experiments over two annotated corpora, KnowledgeNet and FewRel, demonstrate the improved accuracy of our enriched models.
  arXiv  Detail & Related papers  (2022-12-19T11:26:23Z)
• Long Document Summarization with Top-down and Bottom-up Inference [113.29319668246407]
  We propose a principled inference framework to improve summarization models on two aspects. Our framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency. We demonstrate the effectiveness of the proposed framework on a diverse set of summarization datasets.
  arXiv  Detail & Related papers  (2022-03-15T01:24:51Z)
• A Study on Token Pruning for ColBERT [0.7646713951724011]
  The ColBERT model has recently been proposed as an effective BERT based ranker. The big downside of the model is the index size, which scales linearly with the number of tokens in the collection. In this paper, we study various designs for ColBERT models in order to attack this problem.
  arXiv  Detail & Related papers  (2021-12-13T10:24:54Z)
• ColBERTv2: Effective and Efficient Retrieval via Lightweight Late Interaction [15.336103841957328]
  ColBERTv2 is a retriever that couples an aggressive residual compression mechanism with a denoised supervision strategy. We evaluate ColBERTv2 across a range of benchmarks, establishing state-of-the-art quality within and outside the training domain.
  arXiv  Detail & Related papers  (2021-12-02T18:38:50Z)
• Rethinking Space-Time Networks with Improved Memory Coverage for Efficient Video Object Segmentation [68.45737688496654]
  We establish correspondences directly between frames without re-encoding the mask features for every object. With the correspondences, every node in the current query frame is inferred by aggregating features from the past in an associative fashion. We validated that every memory node now has a chance to contribute, and experimentally showed that such diversified voting is beneficial to both memory efficiency and inference accuracy.
  arXiv  Detail & Related papers  (2021-06-09T16:50:57Z)
• Distilling Dense Representations for Ranking using Tightly-Coupled Teachers [52.85472936277762]
  We apply knowledge distillation to improve the recently proposed late-interaction ColBERT model. We distill the knowledge from ColBERT's expressive MaxSim operator for computing relevance scores into a simple dot product. We empirically show that our approach improves query latency and greatly reduces the onerous storage requirements of ColBERT.
  arXiv  Detail & Related papers  (2020-10-22T02:26:01Z)
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT [24.288824715337483]
  ColBERT is a novel ranking model that adapts deep LMs for efficient retrieval. We extensively evaluate ColBERT using two recent passage search datasets.
  arXiv  Detail & Related papers  (2020-04-27T14:21:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.