Ranked List Truncation for Large Language Model-based Re-Ranking

• URL: http://arxiv.org/abs/2404.18185v1
• Date: Sun, 28 Apr 2024 13:39:33 GMT
• Title: Ranked List Truncation for Large Language Model-based Re-Ranking
• Authors: Chuan Meng, Negar Arabzadeh, Arian Askari, Mohammad Aliannejadi, Maarten de Rijke,
• Abstract summary: We study ranked list truncation (RLT) from a novel "retrieve-then-re-rank" perspective, where we optimize re-ranking by truncating the retrieved list. RLT is crucial for re-ranking as it can improve re-ranking efficiency by sending variable-length candidate lists to a re-ranker. We reproduce existing RLT methods in the context of re-ranking, especially newly emerged large language model (LLM)-based re-ranking.
• Score: 53.97064615557883
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We study ranked list truncation (RLT) from a novel "retrieve-then-re-rank" perspective, where we optimize re-ranking by truncating the retrieved list (i.e., trim re-ranking candidates). RLT is crucial for re-ranking as it can improve re-ranking efficiency by sending variable-length candidate lists to a re-ranker on a per-query basis. It also has the potential to improve re-ranking effectiveness. Despite its importance, there is limited research into applying RLT methods to this new perspective. To address this research gap, we reproduce existing RLT methods in the context of re-ranking, especially newly emerged large language model (LLM)-based re-ranking. In particular, we examine to what extent established findings on RLT for retrieval are generalizable to the "retrieve-then-re-rank" setup from three perspectives: (i) assessing RLT methods in the context of LLM-based re-ranking with lexical first-stage retrieval, (ii) investigating the impact of different types of first-stage retrievers on RLT methods, and (iii) investigating the impact of different types of re-rankers on RLT methods. We perform experiments on the TREC 2019 and 2020 deep learning tracks, investigating 8 RLT methods for pipelines involving 3 retrievers and 2 re-rankers. We reach new insights into RLT methods in the context of re-ranking.

Related papers

• Attention in Large Language Models Yields Efficient Zero-Shot Re-Rankers [7.6245627565464]
  Large language models (LLMs) have become popular choices for zero-shot re-ranking in IR systems. We propose in-context re-ranking (ICR), a novel method that leverages the change in attention pattern caused by the search query for accurate and efficient re-ranking. Our findings call for further exploration on novel ways of utilizing open-weight LLMs beyond text generation.
  arXiv  Detail & Related papers  (2024-10-03T16:25:37Z)
• MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models [34.39053202801489]
  In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts. We propose a novel query rewriting method MaFeRw, which improves RAG performance by integrating multi-aspect feedback from both the retrieval process and generated results. Experimental results on two conversational RAG datasets demonstrate that MaFeRw achieves superior generation metrics and more stable training compared to baselines.
  arXiv  Detail & Related papers  (2024-08-30T07:57:30Z)
• Re-Ranking Step by Step: Investigating Pre-Filtering for Re-Ranking with Large Language Models [5.0490573482829335]
  Large Language Models (LLMs) have been revolutionizing a myriad of natural language processing tasks with their diverse zero-shot capabilities. This paper investigates the use of a pre-filtering step before passage re-ranking in information retrieval (IR) Our experiments show that this pre-filtering then allows the LLM to perform significantly better at the re-ranking task.
  arXiv  Detail & Related papers  (2024-06-26T20:12:24Z)
• Prioritized Trajectory Replay: A Replay Memory for Data-driven Reinforcement Learning [52.49786369812919]
  We propose a memory technique, (Prioritized) Trajectory Replay (TR/PTR), which extends the sampling perspective to trajectories. TR enhances learning efficiency by backward sampling of trajectories that optimize the use of subsequent state information. We demonstrate the benefits of integrating TR and PTR with existing offline RL algorithms on D4RL.
  arXiv  Detail & Related papers  (2023-06-27T14:29:44Z)
• Retrieval-Pretrained Transformer: Long-range Language Modeling with Self-retrieval [51.437420003471615]
  We propose the Retrieval-Pretrained Transformer (RPT), an architecture and training procedure for jointly training a retrieval-augmented LM from scratch. RPT improves retrieval quality and subsequently perplexity across the board compared to strong baselines.
  arXiv  Detail & Related papers  (2023-06-23T10:18:02Z)
• Can Continual Learning Improve Long-Tailed Recognition? Toward a Unified Framework [16.457778420360537]
  Long-Tailed Recognition methods aim to accurately learn a dataset comprising both a larger Head set and a smaller Tail set. We show that Continual Learning (CL) methods can effectively update the weights of the learner to learn the Tail without forgetting the Head. We also assess the applicability of CL techniques on real-world data by exploring CL on the naturally imbalanced256 dataset.
  arXiv  Detail & Related papers  (2023-06-23T03:05:33Z)
• Zero-Shot Listwise Document Reranking with a Large Language Model [58.64141622176841]
  We propose Listwise Reranker with a Large Language Model (LRL), which achieves strong reranking effectiveness without using any task-specific training data. Experiments on three TREC web search datasets demonstrate that LRL not only outperforms zero-shot pointwise methods when reranking first-stage retrieval results, but can also act as a final-stage reranker.
  arXiv  Detail & Related papers  (2023-05-03T14:45:34Z)
• Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agents [56.104476412839944]
  Large Language Models (LLMs) have demonstrated remarkable zero-shot generalization across various language-related tasks. This paper investigates generative LLMs for relevance ranking in Information Retrieval (IR) To address concerns about data contamination of LLMs, we collect a new test set called NovelEval. To improve efficiency in real-world applications, we delve into the potential for distilling the ranking capabilities of ChatGPT into small specialized models.
  arXiv  Detail & Related papers  (2023-04-19T10:16:03Z)
• LaPraDoR: Unsupervised Pretrained Dense Retriever for Zero-Shot Text Retrieval [55.097573036580066]
  Experimental results show that LaPraDoR achieves state-of-the-art performance compared with supervised dense retrieval models. Compared to re-ranking, our lexicon-enhanced approach can be run in milliseconds (22.5x faster) while achieving superior performance.
  arXiv  Detail & Related papers  (2022-03-11T18:53:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.