Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA

• URL: http://arxiv.org/abs/2604.23336v1
• Date: Sat, 25 Apr 2026 14:45:33 GMT
• Title: Efficient Rationale-based Retrieval: On-policy Distillation from Generative Rerankers based on JEPA
• Abstract summary: rationale-based retrieval typically requires cross-encoding of query-document pairs.<n>Rabtriever encodes queries and documents, while providing comparable cross query-document comprehension capabilities to rerankers.<n>Rabtriever generalizes well on traditional retrieval benchmarks such as MS MARCO and BEIR.
• Score: 8.95939511590498
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Unlike traditional fact-based retrieval, rationale-based retrieval typically necessitates cross-encoding of query-document pairs using large language models, incurring substantial computational costs. To address this limitation, we propose Rabtriever, which independently encodes queries and documents, while providing comparable cross query-document comprehension capabilities to rerankers. We start from training a LLM-based generative reranker, which puts the document prior to the query and prompts the LLM to generate the relevance score by log probabilities. We then employ it as the teacher of an on-policy distillation framework, with Rabtriever as the student to reconstruct the teacher's contextual-aware query embedding. To achieve this effect, Rabtriever is first initialized from the teacher, with parameters frozen. The Joint-Embedding Predictive Architecture (JEPA) paradigm is then adopted, which integrates a lightweight, trainable predictor between LLM layers and heads, projecting the query embedding into a new hidden space, with the document embedding as the latent vector. JEPA then minimizes the distribution difference between this projected embedding and the teacher embedding. To strengthen the sampling efficiency of on-policy distillation, we also add an auxiliary loss on the reverse KL of LLM logits, to reshape the student's logit distribution. Rabtriever optimizes the teacher's quadratic complexity on the document length to linear, verified both theoretically and empirically. Experiments show that Rabtriever outperforms different retriever baselines across diverse rationale-based tasks, including empathetic conversations and robotic manipulations, with minor accuracy degradation from the reranker. Rabtriever also generalizes well on traditional retrieval benchmarks such as MS MARCO and BEIR, with comparable performance to the best retriever baseline.

Related papers

• AlignCoder: Aligning Retrieval with Target Intent for Repository-Level Code Completion [55.21541958868449]
  We propose AlignCoder, a repository-level code completion framework.<n>Our framework generates an enhanced query that bridges the semantic gap between the initial query and the target code.<n>We employ reinforcement learning to train an AlignRetriever that learns to leverage inference information in the enhanced query for more accurate retrieval.
  arXiv  Detail & Related papers  (2026-01-27T15:23:14Z)
• Rethinking On-policy Optimization for Query Augmentation [49.87723664806526]
  We present the first systematic comparison of prompting-based and RL-based query augmentation across diverse benchmarks.<n>We introduce a novel hybrid method, On-policy Pseudo-document Query Expansion (OPQE), which learns to generate a pseudo-document that maximizes retrieval performance.
  arXiv  Detail & Related papers  (2025-10-20T04:16:28Z)
• Reasoning-enhanced Query Understanding through Decomposition and Interpretation [87.56450566014625]
  ReDI is a Reasoning-enhanced approach for query understanding through Decomposition and Interpretation.<n>We compiled a large-scale dataset of real-world complex queries from a major search engine.<n> Experiments on BRIGHT and BEIR demonstrate that ReDI consistently surpasses strong baselines in both sparse and dense retrieval paradigms.
  arXiv  Detail & Related papers  (2025-09-08T10:58:42Z)
• Harnessing the Power of Reinforcement Learning for Language-Model-Based Information Retriever via Query-Document Co-Augmentation [35.70731674603417]
  Large Language Models (LLMs) can be used to augment both user queries and corpus documents.<n>We present an LLM-based retriever empowered to augment both user queries and corpus documents.<n>Our approach significantly enhances LLM-based retrieval performance in both sparse and dense settings.
  arXiv  Detail & Related papers  (2025-06-23T14:14:43Z)
• Utility-Focused LLM Annotation for Retrieval and Retrieval-Augmented Generation [96.18720164390699]
  This paper explores the use of large language models (LLMs) for annotating document utility in training retrieval and retrieval-augmented generation (RAG) systems.<n>Our results show that LLM-generated annotations enhance out-of-domain retrieval performance and improve RAG outcomes compared to models trained solely on human annotations or downstream QA metrics.
  arXiv  Detail & Related papers  (2025-04-07T16:05:52Z)
• Rank-R1: Enhancing Reasoning in LLM-based Document Rerankers via Reinforcement Learning [76.50690734636477]
  We introduce Rank-R1, a novel LLM-based reranker that performs reasoning over both the user query and candidate documents before performing the ranking task.<n>Our experiments on the TREC DL and BRIGHT datasets show that Rank-R1 is highly effective, especially for complex queries.
  arXiv  Detail & Related papers  (2025-03-08T03:14:26Z)
• Large Language Model Can Be a Foundation for Hidden Rationale-Based Retrieval [12.83513794686623]
  In this paper, we propose and study a more challenging type of retrieval task, called hidden rationale retrieval.<n>To address such problems, an instruction-tuned Large language model (LLM) with a cross-encoder architecture could be a reasonable choice.<n>We name this retrieval framework by RaHoRe and verify its zero-shot and fine-tuned performance superiority on Emotional Support Conversation (ESC)
  arXiv  Detail & Related papers  (2024-12-21T13:19:15Z)
• Bridging Relevance and Reasoning: Rationale Distillation in Retrieval-Augmented Generation [45.366826050955105]
  We propose RADIO, a novel and practical preference alignment framework with RAtionale DIstillatiOn.<n>We first propose a rationale extraction method that leverages the reasoning capabilities of Large Language Models (LLMs) to extract the rationales necessary for answering the query.<n> Subsequently, a rationale-based alignment process is designed to rerank the documents based on the extracted rationales, and fine-tune the reranker to align the preferences.
  arXiv  Detail & Related papers  (2024-12-11T16:32:41Z)
• Query Rewriting for Retrieval-Augmented Large Language Models [139.242907155883]
  Large Language Models (LLMs) play powerful, black-box readers in the retrieve-then-read pipeline. This work introduces a new framework, Rewrite-Retrieve-Read instead of the previous retrieve-then-read for the retrieval-augmented LLMs.
  arXiv  Detail & Related papers  (2023-05-23T17:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.