Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model

• URL: http://arxiv.org/abs/2409.17745v3
• Date: Fri, 4 Oct 2024 18:35:14 GMT
• Title: Few-shot Prompting for Pairwise Ranking: An Effective Non-Parametric Retrieval Model
• Authors: Nilanjan Sinhababu, Andrew Parry, Debasis Ganguly, Debasis Samanta, Pabitra Mitra,
• Abstract summary: We propose a pairwise few-shot ranker that achieves a close performance to that of a supervised model without requiring any complex training pipeline. Our method also achieves a close performance to that of a supervised model without requiring any complex training pipeline.
• Score: 18.111868378615206
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A supervised ranking model, despite its advantage of being effective, usually involves complex processing - typically multiple stages of task-specific pre-training and fine-tuning. This has motivated researchers to explore simpler pipelines leveraging large language models (LLMs) that are capable of working in a zero-shot manner. However, since zero-shot inference does not make use of a training set of pairs of queries and their relevant documents, its performance is mostly worse than that of supervised models, which are trained on such example pairs. Motivated by the existing findings that training examples generally improve zero-shot performance, in our work, we explore if this also applies to ranking models. More specifically, given a query and a pair of documents, the preference prediction task is improved by augmenting examples of preferences for similar queries from a training set. Our proposed pairwise few-shot ranker demonstrates consistent improvements over the zero-shot baseline on both in-domain (TREC DL) and out-domain (BEIR subset) retrieval benchmarks. Our method also achieves a close performance to that of a supervised model without requiring any complex training pipeline.

Related papers

• Enhancing Retrieval Performance: An Ensemble Approach For Hard Negative Mining [0.0]
  This study focuses on explaining the crucial role of hard negatives in the training process of cross-encoder models. We have developed a robust hard negative mining technique for efficient training of cross-encoder re-rank models on an enterprise dataset.
  arXiv  Detail & Related papers  (2024-10-18T05:23:39Z)
• Understanding prompt engineering may not require rethinking generalization [56.38207873589642]
  We show that the discrete nature of prompts, combined with a PAC-Bayes prior given by a language model, results in generalization bounds that are remarkably tight by the standards of the literature. This work provides a possible justification for the widespread practice of prompt engineering.
  arXiv  Detail & Related papers  (2023-10-06T00:52:48Z)
• Meta-training with Demonstration Retrieval for Efficient Few-shot Learning [11.723856248352007]
  Large language models show impressive results on few-shot NLP tasks. These models are memory and computation-intensive. We propose meta-training with demonstration retrieval.
  arXiv  Detail & Related papers  (2023-06-30T20:16:22Z)
• Single-Stage Visual Relationship Learning using Conditional Queries [60.90880759475021]
  TraCQ is a new formulation for scene graph generation that avoids the multi-task learning problem and the entity pair distribution. We employ a DETR-based encoder-decoder conditional queries to significantly reduce the entity label space as well. Experimental results show that TraCQ not only outperforms existing single-stage scene graph generation methods, it also beats many state-of-the-art two-stage methods on the Visual Genome dataset.
  arXiv  Detail & Related papers  (2023-06-09T06:02:01Z)
• Zero-shot Task Preference Addressing Enabled by Imprecise Bayesian Continual Learning [19.11678487931003]
  We propose Imprecise Bayesian Continual Learning (IBCL) to address preferences on task-performance trade-offs. IBCL does not require any additional training overhead to construct preference-addressing models from its knowledge base. We show that models obtained by IBCL have guarantees in identifying the preferred parameters.
  arXiv  Detail & Related papers  (2023-05-24T06:39:00Z)
• Assessing Out-of-Domain Language Model Performance from Few Examples [38.245449474937914]
  We address the task of predicting out-of-domain (OOD) performance in a few-shot fashion. We benchmark the performance on this task when looking at model accuracy on the few-shot examples. We show that attribution-based factors can help rank relative model OOD performance.
  arXiv  Detail & Related papers  (2022-10-13T04:45:26Z)
• Generalization Properties of Retrieval-based Models [50.35325326050263]
  Retrieval-based machine learning methods have enjoyed success on a wide range of problems. Despite growing literature showcasing the promise of these models, the theoretical underpinning for such models remains underexplored. We present a formal treatment of retrieval-based models to characterize their generalization ability.
  arXiv  Detail & Related papers  (2022-10-06T00:33:01Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• On the Role of Supervision in Unsupervised Constituency Parsing [59.55128879760495]
  A few-shot parsing approach can outperform all the unsupervised parsing methods by a significant margin. This suggests that, in order to arrive at fair conclusions, we should carefully consider the amount of labeled data used for model development.
  arXiv  Detail & Related papers  (2020-10-06T01:34:58Z)
• Document Ranking with a Pretrained Sequence-to-Sequence Model [56.44269917346376]
  We show how a sequence-to-sequence model can be trained to generate relevance labels as "target words" Our approach significantly outperforms an encoder-only model in a data-poor regime.
  arXiv  Detail & Related papers  (2020-03-14T22:29:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.