Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG

• URL: http://arxiv.org/abs/2602.03645v1
• Date: Tue, 03 Feb 2026 15:30:14 GMT
• Title: Reinforcement Fine-Tuning for History-Aware Dense Retriever in RAG
• Authors: Yicheng Zhang, Zhen Qin, Zhaomin Wu, Wenqi Zhang, Shuiguang Deng,
• Abstract summary: Retrieval-augmented generation (RAG) enables large language models to produce evidence-based responses.<n>Existing solutions suffer from objective mismatch between retriever optimization and the goal of RAG pipeline.
• Score: 29.46121429194507
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Retrieval-augmented generation (RAG) enables large language models (LLMs) to produce evidence-based responses, and its performance hinges on the matching between the retriever and LLMs. Retriever optimization has emerged as an efficient alternative to fine-tuning LLMs. However, existing solutions suffer from objective mismatch between retriever optimization and the goal of RAG pipeline. Reinforcement learning (RL) provides a promising solution to address this limitation, yet applying RL to retriever optimization introduces two fundamental challenges: 1) the deterministic retrieval is incompatible with RL formulations, and 2) state aliasing arises from query-only retrieval in multi-hop reasoning. To address these challenges, we replace deterministic retrieval with stochastic sampling and formulate RAG as a Markov decision process, making retriever optimizable by RL. Further, we incorporate retrieval history into the state at each retrieval step to mitigate state aliasing. Extensive experiments across diverse RAG pipelines, datasets, and retriever scales demonstrate consistent improvements of our approach in RAG performance.

Related papers

• ReCALL: Recalibrating Capability Degradation for MLLM-based Composed Image Retrieval [64.14282916266998]
  Composed Image Retrieval aims to retrieve target images based on a hybrid query comprising a reference image and a modification text.<n>We propose ReCALL, a model-agnostic framework that follows a diagnose-generate-refine pipeline.<n>Experiments on CIRR and FashionIQ show that ReCALL consistently recalibrates degraded capabilities and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-02-02T04:52:54Z)
• Rethinking On-policy Optimization for Query Augmentation [49.87723664806526]
  We present the first systematic comparison of prompting-based and RL-based query augmentation across diverse benchmarks.<n>We introduce a novel hybrid method, On-policy Pseudo-document Query Expansion (OPQE), which learns to generate a pseudo-document that maximizes retrieval performance.
  arXiv  Detail & Related papers  (2025-10-20T04:16:28Z)
• Generalized Reinforcement Learning for Retriever-Specific Query Rewriter with Unstructured Real-World Documents [4.200973008100858]
  textbfRL-QR is a reinforcement learning framework for retriever-specific query rewriting.<n> RL-QR trains query rewriters tailored to specific retrievers, enhancing retrieval performance across varied domains.<n>Our findings highlight RL-QR's potential to revolutionize query optimization for RAG systems.
  arXiv  Detail & Related papers  (2025-07-31T04:55:21Z)
• LTRR: Learning To Rank Retrievers for LLMs [53.285436927963865]
  We show that routing-based RAG systems can outperform the best single-retriever-based systems.<n>Performance gains are especially pronounced in models trained with the Answer Correctness (AC) metric.<n>As part of the SIGIR 2025 LiveRAG challenge, our submitted system demonstrated the practical viability of our approach.
  arXiv  Detail & Related papers  (2025-06-16T17:53:18Z)
• Direct Retrieval-augmented Optimization: Synergizing Knowledge Selection and Language Models [83.8639566087953]
  We propose a direct retrieval-augmented optimization framework, named DRO, that enables end-to-end training of two key components.<n>DRO alternates between two phases: (i) document permutation estimation and (ii) re-weighted, progressively improving RAG components.<n>Our theoretical analysis reveals that DRO is analogous to policy-gradient methods in reinforcement learning.
  arXiv  Detail & Related papers  (2025-05-05T23:54:53Z)
• LevelRAG: Enhancing Retrieval-Augmented Generation with Multi-hop Logic Planning over Rewriting Augmented Searchers [24.01783076521377]
  Retrieval-Augmented Generation (RAG) is a crucial method for mitigating hallucinations in Large Language Models (LLMs)<n>Existing RAG methods typically employ query rewriting to clarify the user intent and manage multi-hop logic, while using hybrid retrieval to expand search scope.<n>We introduce a high-level searcher that decomposes complex queries into atomic queries, independent of any retriever-specific optimizations.<n>To harness the strengths of sparse retrievers for precise keyword retrieval, we have developed a new sparse searcher that employs Lucene syntax to enhance retrieval accuracy.
  arXiv  Detail & Related papers  (2025-02-25T12:09:16Z)
• Chain-of-Retrieval Augmented Generation [91.02950964802454]
  This paper introduces an approach for training o1-like RAG models that retrieve and reason over relevant information step by step before generating the final answer.<n>Our proposed method, CoRAG, allows the model to dynamically reformulate the query based on the evolving state.
  arXiv  Detail & Related papers  (2025-01-24T09:12:52Z)
• MBA-RAG: a Bandit Approach for Adaptive Retrieval-Augmented Generation through Question Complexity [30.346398341996476]
  We propose a reinforcement learning-based framework that dynamically selects the most suitable retrieval strategy based on query complexity.<n>Our method achieves new state of the art results on multiple single-hop and multi-hop datasets while reducing retrieval costs.
  arXiv  Detail & Related papers  (2024-12-02T14:55:02Z)
• Toward Optimal Search and Retrieval for RAG [39.69494982983534]
  Retrieval-augmented generation (RAG) is a promising method for addressing some of the memory-related challenges associated with Large Language Models (LLMs) Here, we work towards the goal of understanding how retrievers can be optimized for RAG pipelines for common tasks such as Question Answering (QA)
  arXiv  Detail & Related papers  (2024-11-11T22:06:51Z)
• Learning to Retrieve Iteratively for In-Context Learning [56.40100968649039]
  iterative retrieval is a novel framework that empowers retrievers to make iterative decisions through policy optimization. We instantiate an iterative retriever for composing in-context learning exemplars and apply it to various semantic parsing tasks. By adding only 4M additional parameters for state encoding, we convert an off-the-shelf dense retriever into a stateful iterative retriever.
  arXiv  Detail & Related papers  (2024-06-20T21:07:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.