SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval

• URL: http://arxiv.org/abs/2109.10086v1
• Date: Tue, 21 Sep 2021 10:43:42 GMT
• Title: SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval
• Authors: Thibault Formal, Carlos Lassance, Benjamin Piwowarski, St\'ephane Clinchant
• Abstract summary: SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. We modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. Overall, SPLADE is considerably improved with more than $9$% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.
• Score: 11.38022203865326
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: In neural Information Retrieval (IR), ongoing research is directed towards improving the first retriever in ranking pipelines. Learning dense embeddings to conduct retrieval using efficient approximate nearest neighbors methods has proven to work well. Meanwhile, there has been a growing interest in learning \emph{sparse} representations for documents and queries, that could inherit from the desirable properties of bag-of-words models such as the exact matching of terms and the efficiency of inverted indexes. Introduced recently, the SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. In this paper, we build on SPLADE and propose several significant improvements in terms of effectiveness and/or efficiency. More specifically, we modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. We also report results on the BEIR benchmark. Overall, SPLADE is considerably improved with more than $9$\% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.

Related papers

• Monte Carlo Tree Search Boosts Reasoning via Iterative Preference Learning [55.96599486604344]
  We introduce an approach aimed at enhancing the reasoning capabilities of Large Language Models (LLMs) through an iterative preference learning process. We use Monte Carlo Tree Search (MCTS) to iteratively collect preference data, utilizing its look-ahead ability to break down instance-level rewards into more granular step-level signals. The proposed algorithm employs Direct Preference Optimization (DPO) to update the LLM policy using this newly generated step-level preference data.
  arXiv  Detail & Related papers  (2024-05-01T11:10:24Z)
• LLM-Augmented Retrieval: Enhancing Retrieval Models Through Language Models and Doc-Level Embedding [2.0257616108612373]
  This paper introduces a model-agnostic doc-level embedding framework through large language model augmentation. We have been able to significantly improve the effectiveness of widely-used retriever models.
  arXiv  Detail & Related papers  (2024-04-08T19:29:07Z)
• The Languini Kitchen: Enabling Language Modelling Research at Different Scales of Compute [66.84421705029624]
  We introduce an experimental protocol that enables model comparisons based on equivalent compute, measured in accelerator hours. We pre-process an existing large, diverse, and high-quality dataset of books that surpasses existing academic benchmarks in quality, diversity, and document length. This work also provides two baseline models: a feed-forward model derived from the GPT-2 architecture and a recurrent model in the form of a novel LSTM with ten-fold throughput.
  arXiv  Detail & Related papers  (2023-09-20T10:31:17Z)
• Consensus-Adaptive RANSAC [104.87576373187426]
  We propose a new RANSAC framework that learns to explore the parameter space by considering the residuals seen so far via a novel attention layer. The attention mechanism operates on a batch of point-to-model residuals, and updates a per-point estimation state to take into account the consensus found through a lightweight one-step transformer.
  arXiv  Detail & Related papers  (2023-07-26T08:25:46Z)
• SPRINT: A Unified Toolkit for Evaluating and Demystifying Zero-shot Neural Sparse Retrieval [92.27387459751309]
  We provide SPRINT, a unified Python toolkit for evaluating neural sparse retrieval. We establish strong and reproducible zero-shot sparse retrieval baselines across the well-acknowledged benchmark, BEIR. We show that SPLADEv2 produces sparse representations with a majority of tokens outside of the original query and document.
  arXiv  Detail & Related papers  (2023-07-19T22:48:02Z)
• CascadER: Cross-Modal Cascading for Knowledge Graph Link Prediction [22.96768147978534]
  We propose a tiered ranking architecture CascadER to maintain the ranking accuracy of full ensembling while improving efficiency considerably. CascadER uses LMs to rerank the outputs of more efficient base KGEs, relying on an adaptive subset selection scheme aimed at invoking the LMs minimally while maximizing accuracy gain over the KGE. Our empirical analyses reveal that diversity of models across modalities and preservation of individual models' confidence signals help explain the effectiveness of CascadER.
  arXiv  Detail & Related papers  (2022-05-16T22:55:45Z)
• From Distillation to Hard Negative Sampling: Making Sparse Neural IR Models More Effective [15.542082655342476]
  We build on SPLADE -- a sparse expansion-based retriever -- and show to which extent it is able to benefit from the same training improvements as dense models. We study the link between effectiveness and efficiency, on in-domain and zero-shot settings.
  arXiv  Detail & Related papers  (2022-05-10T08:08:43Z)
• Curriculum Learning for Dense Retrieval Distillation [20.25741148622744]
  We propose a generic curriculum learning based optimization framework called CL-DRD. CL-DRD controls the difficulty level of training data produced by the re-ranking (teacher) model. Experiments on three public passage retrieval datasets demonstrate the effectiveness of our proposed framework.
  arXiv  Detail & Related papers  (2022-04-28T17:42:21Z)
• LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval [55.097573036580066]
  Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models. Compared to re-ranking, our lexicon-enhanced approach can be run in milliseconds (22.5x faster) while achieving superior performance.
  arXiv  Detail & Related papers  (2022-03-11T18:53:12Z)
• Enhancing the Generalization for Intent Classification and Out-of-Domain Detection in SLU [70.44344060176952]
  Intent classification is a major task in spoken language understanding (SLU) Recent works have shown that using extra data and labels can improve the OOD detection performance. This paper proposes to train a model with only IND data while supporting both IND intent classification and OOD detection.
  arXiv  Detail & Related papers  (2021-06-28T08:27:38Z)
• An Improved Baseline for Sentence-level Relation Extraction [17.50856935207308]
  Sentence-level relation extraction (RE) aims at identifying the relationship between two entities in a sentence. In this paper, we revisit two aspects of RE models that are not thoroughly studied, namely entity representation and NA instance prediction.
  arXiv  Detail & Related papers  (2021-02-02T07:57:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.