InstUPR : Instruction-based Unsupervised Passage Reranking with Large Language Models

• URL: http://arxiv.org/abs/2403.16435v1
• Date: Mon, 25 Mar 2024 05:31:22 GMT
• Title: InstUPR : Instruction-based Unsupervised Passage Reranking with Large Language Models
• Authors: Chao-Wei Huang, Yun-Nung Chen,
• Abstract summary: InstUPR is an unsupervised passage reranking method based on large language models (LLMs) We introduce a soft score aggregation technique and employ pairwise reranking for unsupervised passage reranking. Experiments on the BEIR benchmark demonstrate that InstUPR outperforms unsupervised baselines as well as an instruction-tuned reranker.
• Score: 35.067998820937284
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: This paper introduces InstUPR, an unsupervised passage reranking method based on large language models (LLMs). Different from existing approaches that rely on extensive training with query-document pairs or retrieval-specific instructions, our method leverages the instruction-following capabilities of instruction-tuned LLMs for passage reranking without any additional fine-tuning. To achieve this, we introduce a soft score aggregation technique and employ pairwise reranking for unsupervised passage reranking. Experiments on the BEIR benchmark demonstrate that InstUPR outperforms unsupervised baselines as well as an instruction-tuned reranker, highlighting its effectiveness and superiority. Source code to reproduce all experiments is open-sourced at https://github.com/MiuLab/InstUPR

Related papers

• FIRST: Faster Improved Listwise Reranking with Single Token Decoding [56.727761901751194]
  First, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.
  arXiv  Detail & Related papers  (2024-06-21T21:27:50Z)
• Disperse-Then-Merge: Pushing the Limits of Instruction Tuning via Alignment Tax Reduction [75.25114727856861]
  Large language models (LLMs) tend to suffer from deterioration at the latter stage ofSupervised fine-tuning process. We introduce a simple disperse-then-merge framework to address the issue. Our framework outperforms various sophisticated methods such as data curation and training regularization on a series of standard knowledge and reasoning benchmarks.
  arXiv  Detail & Related papers  (2024-05-22T08:18:19Z)
• Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankers [56.12593882838412]
  We introduce a novel instruction distillation method to rank documents. We first rank documents using the effective pairwise approach with complex instructions, and then distill the teacher predictions to the pointwise approach with simpler instructions. Our approach surpasses the performance of existing supervised methods like monoT5 and is on par with the state-of-the-art zero-shot methods.
  arXiv  Detail & Related papers  (2023-11-02T19:16:21Z)
• Tuna: Instruction Tuning using Feedback from Large Language Models [74.04950416204551]
  We propose finetuning an instruction-tuned large language model using our novel textitprobabilistic ranking and textitcontextual ranking approaches. Probabilistic ranking enables the instruction-tuned model to inherit the relative rankings of high-quality and low-quality responses from the teacher LLM. On the other hand, learning with contextual ranking allows the model to refine its own response distribution using the contextual understanding ability of stronger LLMs.
  arXiv  Detail & Related papers  (2023-10-20T09:55:06Z)
• Reflection-Tuning: Data Recycling Improves LLM Instruction-Tuning [79.32236399694077]
  Low-quality data in the training set are usually detrimental to instruction tuning. We propose a novel method, termed "reflection-tuning" This approach utilizes an oracle LLM to recycle the original training data by introspecting and enhancing the quality of instructions and responses in the data.
  arXiv  Detail & Related papers  (2023-10-18T05:13:47Z)
• DoG-Instruct: Towards Premium Instruction-Tuning Data via Text-Grounded Instruction Wrapping [41.89443082174044]
  This paper proposes a scalable solution to the problem of finding high-quality instruction-response pairs. It involves training LLMs to generate pairs based on human-written documents, rather than relying solely on self-generation without context. Our proposed method not only exploits the advantages of human-written documents in reducing hallucinations but also utilizes an LLM to wrap the expression of documents.
  arXiv  Detail & Related papers  (2023-09-11T13:41:18Z)
• ReLLa: Retrieval-enhanced Large Language Models for Lifelong Sequential Behavior Comprehension in Recommendation [43.270424225285105]
  We focus on adapting and empowering a pure large language model for zero-shot and few-shot recommendation tasks. We propose Retrieval-enhanced Large Language models (ReLLa) for recommendation tasks in both zero-shot and few-shot settings.
  arXiv  Detail & Related papers  (2023-08-22T02:25:04Z)
• AugTriever: Unsupervised Dense Retrieval and Domain Adaptation by Scalable Data Augmentation [44.93777271276723]
  We propose two approaches that enable annotation-free and scalable training by creating pseudo querydocument pairs. The query extraction method involves selecting salient spans from the original document to generate pseudo queries. The transferred query generation method utilizes generation models trained for other NLP tasks, such as summarization, to produce pseudo queries.
  arXiv  Detail & Related papers  (2022-12-17T10:43:25Z)
• Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models [63.808843089941405]
  Large pretrained LanguageModels (LMs) generate text with remarkable quality, but only sequentially from left to right. We present Reflective Decoding, a novel unsupervised algorithm that allows for direct application of unidirectional LMs to non-sequential tasks. Our 2-step approach requires no supervision or even parallel corpora, only two off-the-shelf pretrained LMs in opposite directions.
  arXiv  Detail & Related papers  (2020-10-16T18:02:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.