Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?

• URL: http://arxiv.org/abs/2605.27881v2
• Date: Fri, 29 May 2026 02:01:20 GMT
• Title: Retrieval, Reward, and Training Protocols: What Matters in Training Search Agents?
• Abstract summary: We present a controlled empirical study that isolates three under-explored dimensions of search agent training.<n>First, we identify a critical data-coverage issue in the widely used Wikipedia 2018 corpus and show that correcting it alone yields larger gains than the differences between training algorithms.<n>Second, we systematically compare outcome-based and process-based reward methods across three base models, finding that the simplest outcome-based approach achieves competitive or superior performance in most settings.<n>Third, we analyze training data diversity, off-policy data utilization, and search budget scaling, distilling practical guidelines for training effective search agents.
• Score: 15.605151408619895
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Search agents powered by large language models can autonomously decompose queries, retrieve information, and synthesize answers through multi-step reasoning. However, the rapid growth of training methods has outpaced controlled comparison: existing works differ in retrieval corpora, reward designs, and training protocols, making it unclear what actually drives improvements. We present a controlled empirical study that isolates three under-explored dimensions of search agent training. First, we identify a critical data-coverage issue in the widely used Wikipedia 2018 corpus and show that correcting it alone yields larger gains than the differences between training algorithms. Second, we systematically compare outcome-based and process-based reward methods across three base models, finding that the simplest outcome-based approach achieves competitive or superior performance in most settings, and that process-level credit assignment can over-correct agent behavior. Third, we analyze training data diversity, off-policy data utilization, and search budget scaling, distilling practical guidelines for training effective search agents. Our code is available at https://github.com/YiboZhao624/SearchAgentReview.

Related papers

• $S^3$-R1: Learning to Retrieve and Answer Step-by-Step with Synthetic Data [2.003941363902692]
  We introduce S3-R1 (Synthetic data and stabilized Search R1), a framework that couples a datacentric approach with denser learning signals.<n>We first develop a synthetic generation and pipeline that derives diverse, multi-hop questions from existing documents.<n>We then pair this expanded training set with a reward structure that evaluates both intermediate search quality and the correctness final answer.
  arXiv  Detail & Related papers  (2026-05-02T05:01:05Z)
• CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search [51.911048955965136]
  CoSearch is a framework that jointly trains a multi-step reasoning agent and a generative document ranking model.<n>Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant.
  arXiv  Detail & Related papers  (2026-04-19T17:48:17Z)
• PRAISE: Prefix-Based Rollout Reuse in Agentic Search Training [28.912613644535668]
  We present PRAISE, a framework for improving data efficiency and credit assignment in agentic search training.<n>Our method uses a single shared model for both search policy learning and prefix answer evaluation.<n>Experiments on multi-hop QA benchmarks show that PRAISE consistently improves performance over strong baselines.
  arXiv  Detail & Related papers  (2026-04-04T10:23:46Z)
• Learning to Retrieve from Agent Trajectories [72.8923565916533]
  We argue that retrieval models for agentic search should be trained directly from agent interaction data.<n>We introduce learning to retrieve from agent trajectories as a new training paradigm, where supervision is derived from multi-step agent interactions.<n>Our results highlight agent trajectories as a practical and scalable supervision source, pointing to a promising direction for retrieval in the era of agentic search.
  arXiv  Detail & Related papers  (2026-03-30T17:59:02Z)
• KARL: Knowledge Agents via Reinforcement Learning [63.627906947205624]
  We present a system for training enterprise search agents via reinforcement learning.<n> KARLBench is a multi-capability evaluation suite spanning six distinct search regimes.<n>We show that models trained across heterogeneous search behaviors generalize substantially better than those optimized for any single benchmark.
  arXiv  Detail & Related papers  (2026-03-05T14:30:25Z)
• Divide and Conquer: Hybrid Pre-training for Person Search [40.13016375392472]
  We propose a hybrid pre-training framework specifically designed for person search using sub-task data only. Our model can achieve significant improvements across diverse protocols, such as person search method, fine-tuning data, pre-training data and model backbone. Our code and pre-trained models are released for plug-and-play usage to the person search community.
  arXiv  Detail & Related papers  (2023-12-13T08:33:50Z)
• Back to Basics: A Simple Recipe for Improving Out-of-Domain Retrieval in Dense Encoders [63.28408887247742]
  We study whether training procedures can be improved to yield better generalization capabilities in the resulting models. We recommend a simple recipe for training dense encoders: Train on MSMARCO with parameter-efficient methods, such as LoRA, and opt for using in-batch negatives unless given well-constructed hard negatives.
  arXiv  Detail & Related papers  (2023-11-16T10:42:58Z)
• Benchmarking Neural Network Training Algorithms [52.890134877995195]
  Training algorithms are an essential part of every deep learning pipeline.<n>As a community, we are unable to reliably identify training algorithm improvements.<n>We introduce a new, competitive, time-to-result benchmark using multiple workloads running on fixed hardware.
  arXiv  Detail & Related papers  (2023-06-12T15:21:02Z)
• Retrieval-Augmented Reinforcement Learning [63.32076191982944]
  We train a network to map a dataset of past experiences to optimal behavior. The retrieval process is trained to retrieve information from the dataset that may be useful in the current context. We show that retrieval-augmented R2D2 learns significantly faster than the baseline R2D2 agent and achieves higher scores.
  arXiv  Detail & Related papers  (2022-02-17T02:44:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.