RaFe: Ranking Feedback Improves Query Rewriting for RAG

• URL: http://arxiv.org/abs/2405.14431v1
• Date: Thu, 23 May 2024 11:00:19 GMT
• Title: RaFe: Ranking Feedback Improves Query Rewriting for RAG
• Authors: Shengyu Mao, Yong Jiang, Boli Chen, Xiao Li, Peng Wang, Xinyu Wang, Pengjun Xie, Fei Huang, Huajun Chen, Ningyu Zhang,
• Abstract summary: We propose a framework for training query rewriting models free of annotations. By leveraging a publicly available reranker, oursprovides feedback aligned well with the rewriting objectives.
• Score: 83.24385658573198
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As Large Language Models (LLMs) and Retrieval Augmentation Generation (RAG) techniques have evolved, query rewriting has been widely incorporated into the RAG system for downstream tasks like open-domain QA. Many works have attempted to utilize small models with reinforcement learning rather than costly LLMs to improve query rewriting. However, current methods require annotations (e.g., labeled relevant documents or downstream answers) or predesigned rewards for feedback, which lack generalization, and fail to utilize signals tailored for query rewriting. In this paper, we propose ours, a framework for training query rewriting models free of annotations. By leveraging a publicly available reranker, ours~provides feedback aligned well with the rewriting objectives. Experimental results demonstrate that ours~can obtain better performance than baselines.

Related papers

• MaFeRw: Query Rewriting with Multi-Aspect Feedbacks for Retrieval-Augmented Large Language Models [34.39053202801489]
  In a real-world RAG system, the current query often involves spoken ellipses and ambiguous references from dialogue contexts. We propose a novel query rewriting method MaFeRw, which improves RAG performance by integrating multi-aspect feedback from both the retrieval process and generated results. Experimental results on two conversational RAG datasets demonstrate that MaFeRw achieves superior generation metrics and more stable training compared to baselines.
  arXiv  Detail & Related papers  (2024-08-30T07:57:30Z)
• RE-AdaptIR: Improving Information Retrieval through Reverse Engineered Adaptation [37.969478059005574]
  Large language models (LLMs) fine-tuned for text-retrieval have demonstrated state-of-the-art results across several information retrieval benchmarks. We explore the effectiveness of extending reverse engineered adaptation to the context of information retrieval.
  arXiv  Detail & Related papers  (2024-06-20T22:28:11Z)
• Adaptive Query Rewriting: Aligning Rewriters through Marginal Probability of Conversational Answers [66.55612528039894]
  AdaQR is a framework for training query rewriting models with limited rewrite annotations from seed datasets and completely no passage label. A novel approach is proposed to assess retriever's preference for these candidates by the probability of answers conditioned on the conversational query.
  arXiv  Detail & Related papers  (2024-06-16T16:09:05Z)
• RQ-RAG: Learning to Refine Queries for Retrieval Augmented Generation [42.82192656794179]
  Large Language Models (LLMs) exhibit remarkable capabilities but are prone to generating inaccurate or hallucinatory responses. This limitation stems from their reliance on vast pretraining datasets, making them susceptible to errors in unseen scenarios. Retrieval-Augmented Generation (RAG) addresses this by incorporating external, relevant documents into the response generation process.
  arXiv  Detail & Related papers  (2024-03-31T08:58:54Z)
• Improving Retrieval for RAG based Question Answering Models on Financial Documents [0.046603287532620746]
  This paper explores the existing constraints of RAG pipelines and introduces methodologies for enhancing text retrieval. It delves into strategies such as sophisticated chunking techniques, query expansion, the incorporation of metadata annotations, the application of re-ranking algorithms, and the fine-tuning of embedding algorithms.
  arXiv  Detail & Related papers  (2024-03-23T00:49:40Z)
• RAFT: Adapting Language Model to Domain Specific RAG [75.63623523051491]
  We present Retrieval Augmented FineTuning (RAFT), a training recipe that improves the model's ability to answer questions in a "openbook" in-domain settings. RAFT accomplishes this by citing the verbatim right sequence from the relevant document that would help answer the question. RAFT consistently improves the model's performance across PubMed, HotpotQA, and Gorilla datasets.
  arXiv  Detail & Related papers  (2024-03-15T09:26:02Z)
• Regression-aware Inference with LLMs [52.764328080398805]
  We show that an inference strategy can be sub-optimal for common regression and scoring evaluation metrics. We propose alternate inference strategies that estimate the Bayes-optimal solution for regression and scoring metrics in closed-form from sampled responses.
  arXiv  Detail & Related papers  (2024-03-07T03:24:34Z)
• Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting [42.35788605017555]
  We propose utilizing large language models (LLMs) as query rewriters. We define four essential properties for well-formed rewrites and incorporate all of them into the instruction. We introduce the role of rewrite editors for LLMs when initial query rewrites are available, forming a "rewrite-then-edit" process.
  arXiv  Detail & Related papers  (2023-10-15T03:04:17Z)
• Query Rewriting for Retrieval-Augmented Large Language Models [139.242907155883]
  Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline. This work introduces a new framework, Rewrite-Retrieve-Read instead of the previous retrieve-then-read for the retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2023-05-23T17:27:50Z)
• Does Recommend-Revise Produce Reliable Annotations? An Analysis on Missing Instances in DocRED [60.39125850987604]
  We show that a textit-revise scheme results in false negative samples and an obvious bias towards popular entities and relations. The relabeled dataset is released to serve as a more reliable test set of document RE models.
  arXiv  Detail & Related papers  (2022-04-17T11:29:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.