A Reproducibility Study of PLAID

• URL: http://arxiv.org/abs/2404.14989v1
• Date: Tue, 23 Apr 2024 12:46:53 GMT
• Title: A Reproducibility Study of PLAID
• Authors: Sean MacAvaney, Nicola Tonellotto,
• Abstract summary: We compare PLAID with an important baseline missing from the paper: re-ranking a lexical system. We find that applying ColBERTv2 as a re-ranker atop an initial pool of BM25 results provides better efficiency-effectiveness trade-offs in low-latency settings. We find that recently proposed modifications to re-ranking that pull in the neighbors of top-scoring documents overcome this limitation.
• Score: 25.86500025007641
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The PLAID (Performance-optimized Late Interaction Driver) algorithm for ColBERTv2 uses clustered term representations to retrieve and progressively prune documents for final (exact) document scoring. In this paper, we reproduce and fill in missing gaps from the original work. By studying the parameters PLAID introduces, we find that its Pareto frontier is formed of a careful balance among its three parameters; deviations beyond the suggested settings can substantially increase latency without necessarily improving its effectiveness. We then compare PLAID with an important baseline missing from the paper: re-ranking a lexical system. We find that applying ColBERTv2 as a re-ranker atop an initial pool of BM25 results provides better efficiency-effectiveness trade-offs in low-latency settings. However, re-ranking cannot reach peak effectiveness at higher latency settings due to limitations in recall of lexical matching and provides a poor approximation of an exhaustive ColBERTv2 search. We find that recently proposed modifications to re-ranking that pull in the neighbors of top-scoring documents overcome this limitation, providing a Pareto frontier across all operational points for ColBERTv2 when evaluated using a well-annotated dataset. Curious about why re-ranking methods are highly competitive with PLAID, we analyze the token representation clusters PLAID uses for retrieval and find that most clusters are predominantly aligned with a single token and vice versa. Given the competitive trade-offs that re-ranking baselines exhibit, this work highlights the importance of carefully selecting pertinent baselines when evaluating the efficiency of retrieval engines.

Related papers

• Towards Competitive Search Relevance For Inference-Free Learned Sparse Retrievers [6.773411876899064]
  inference-free sparse models lag far behind in terms of search relevance when compared to both sparse and dense siamese models. We propose two different approaches for performance improvement. First, we introduce the IDF-aware FLOPS loss, which introduces Inverted Document Frequency (IDF) to the sparsification of representations. We find that it mitigates the negative impact of the FLOPS regularization on search relevance, allowing the model to achieve a better balance between accuracy and efficiency.
  arXiv  Detail & Related papers  (2024-11-07T03:46:43Z)
• Efficient Document Ranking with Learnable Late Interactions [73.41976017860006]
  Cross-Encoder (CE) and Dual-Encoder (DE) models are two fundamental approaches for query-document relevance in information retrieval. To predict relevance, CE models use joint query-document embeddings, while DE models maintain factorized query and document embeddings. Recently, late-interaction models have been proposed to realize more favorable latency-quality tradeoffs, by using a DE structure followed by a lightweight scorer.
  arXiv  Detail & Related papers  (2024-06-25T22:50:48Z)
• Once for Both: Single Stage of Importance and Sparsity Search for Vision Transformer Compression [63.23578860867408]
  We investigate how to integrate the evaluations of importance and sparsity scores into a single stage. We present OFB, a cost-efficient approach that simultaneously evaluates both importance and sparsity scores. Experiments demonstrate that OFB can achieve superior compression performance over state-of-the-art searching-based and pruning-based methods.
  arXiv  Detail & Related papers  (2024-03-23T13:22:36Z)
• Large-scale Point Cloud Registration Based on Graph Matching Optimization [30.92028761652611]
  We propose a underlineGraph underlineMatching underlineOptimization based underlineNetwork. The proposed method has been evaluated on the 3DMatch/3DLoMatch benchmarks and the KITTI benchmark.
  arXiv  Detail & Related papers  (2023-02-12T03:29:35Z)
• Deep Active Ensemble Sampling For Image Classification [8.31483061185317]
  Active learning frameworks aim to reduce the cost of data annotation by actively requesting the labeling for the most informative data points. Some proposed approaches include uncertainty-based techniques, geometric methods, implicit combination of uncertainty-based and geometric approaches. We present an innovative integration of recent progress in both uncertainty-based and geometric frameworks to enable an efficient exploration/exploitation trade-off in sample selection strategy. Our framework provides two advantages: (1) accurate posterior estimation, and (2) tune-able trade-off between computational overhead and higher accuracy.
  arXiv  Detail & Related papers  (2022-10-11T20:20:20Z)
• ReAct: Temporal Action Detection with Relational Queries [84.76646044604055]
  This work aims at advancing temporal action detection (TAD) using an encoder-decoder framework with action queries. We first propose a relational attention mechanism in the decoder, which guides the attention among queries based on their relations. Lastly, we propose to predict the localization quality of each action query at inference in order to distinguish high-quality queries.
  arXiv  Detail & Related papers  (2022-07-14T17:46:37Z)
• Long Document Summarization with Top-down and Bottom-up Inference [113.29319668246407]
  We propose a principled inference framework to improve summarization models on two aspects. Our framework assumes a hierarchical latent structure of a document where the top-level captures the long range dependency. We demonstrate the effectiveness of the proposed framework on a diverse set of summarization datasets.
  arXiv  Detail & Related papers  (2022-03-15T01:24:51Z)
• Open-Set Recognition: A Good Closed-Set Classifier is All You Need [146.6814176602689]
  We show that the ability of a classifier to make the 'none-of-above' decision is highly correlated with its accuracy on the closed-set classes. We use this correlation to boost the performance of the cross-entropy OSR 'baseline' by improving its closed-set accuracy. We also construct new benchmarks which better respect the task of detecting semantic novelty.
  arXiv  Detail & Related papers  (2021-10-12T17:58:59Z)
• SPLADE v2: Sparse Lexical and Expansion Model for Information Retrieval [11.38022203865326]
  SPLADE model provides highly sparse representations and competitive results with respect to state-of-the-art dense and sparse approaches. We modify the pooling mechanism, benchmark a model solely based on document expansion, and introduce models trained with distillation. Overall, SPLADE is considerably improved with more than $9$% gains on NDCG@10 on TREC DL 2019, leading to state-of-the-art results on the BEIR benchmark.
  arXiv  Detail & Related papers  (2021-09-21T10:43:42Z)
• CRACT: Cascaded Regression-Align-Classification for Robust Visual Tracking [97.84109669027225]
  We introduce an improved proposal refinement module, Cascaded Regression-Align- Classification (CRAC) CRAC yields new state-of-the-art performances on many benchmarks. In experiments on seven benchmarks including OTB-2015, UAV123, NfS, VOT-2018, TrackingNet, GOT-10k and LaSOT, our CRACT exhibits very promising results in comparison with state-of-the-art competitors.
  arXiv  Detail & Related papers  (2020-11-25T02:18:33Z)
• ColBERT: Efficient and Effective Passage Search via Contextualized Late Interaction over BERT [24.288824715337483]
  ColBERT is a novel ranking model that adapts deep LMs for efficient retrieval. We extensively evaluate ColBERT using two recent passage search datasets.
  arXiv  Detail & Related papers  (2020-04-27T14:21:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.