PRAISE: Prefix-Based Rollout Reuse in Agentic Search Training

• URL: http://arxiv.org/abs/2604.03675v1
• Date: Sat, 04 Apr 2026 10:23:46 GMT
• Title: PRAISE: Prefix-Based Rollout Reuse in Agentic Search Training
• Abstract summary: We present PRAISE, a framework for improving data efficiency and credit assignment in agentic search training.<n>Our method uses a single shared model for both search policy learning and prefix answer evaluation.<n>Experiments on multi-hop QA benchmarks show that PRAISE consistently improves performance over strong baselines.
• Score: 28.912613644535668
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In agentic search, large language models (LLMs) are trained to perform multi-turn retrieval and reasoning for complex tasks such as multi-hop question answering (QA). However, current search-based Reinforcement Learning (RL) methods suffer from two core limitations: expensive long-horizon rollouts are under-utilized during training, and supervision is typically available only at the final answer, resulting in severe reward sparsity. We present Prefix-based Rollout reuse for Agentic search with Intermediate Step rEwards (PRAISE), a framework for improving both data efficiency and credit assignment in agentic search training. Given a complete search trajectory, PRAISE extracts prefix states at different search turns, elicits intermediate answers from them, and uses these prefixes both to construct additional training trajectories and to derive step-level rewards from performance differences across prefixes. Our method uses a single shared model for both search policy learning and prefix answer evaluation, enabling joint optimization without extra human annotations or a separate reward model. Experiments on multi-hop QA benchmarks show that PRAISE consistently improves performance over strong baselines.

Related papers

• SPADER: Step-wise Peer Advantage with Diversity-Aware Exploration Rewards for Multi-Answer Question Answering [28.469701396085952]
  Large language models are increasingly deployed as tool-augmented agents to acquire information beyond parametric knowledge.<n>We propose SPADER, a reinforcement learning framework for long-horizon tool use in Multi-Answer QA.<n>We show that SPADER generally improves recall and overall F1 over prompting-based agents, outcome-supervised RL methods, and recent step-level supervision approaches.
  arXiv  Detail & Related papers  (2026-05-30T07:47:42Z)
• CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search [51.911048955965136]
  CoSearch is a framework that jointly trains a multi-step reasoning agent and a generative document ranking model.<n>Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant.
  arXiv  Detail & Related papers  (2026-04-19T17:48:17Z)
• SubSearch: Intermediate Rewards for Unsupervised Guided Reasoning in Complex Retrieval [61.73547199231695]
  SubSearch is a specialized framework that shifts from outcome-only supervision to intermediate reward signals.<n> Experiments show that rewarding intermediate reasoning steps with intrinsic rewards leads to more robust reasoning traces.<n>SubSearch can help in building reasoning traces that allow agents to better integrate search engines for complex query answering.
  arXiv  Detail & Related papers  (2026-04-08T13:09:47Z)
• APEX-Searcher: Augmenting LLMs' Search Capabilities through Agentic Planning and Execution [10.643859464982071]
  APEX-Searcher is a novel Agentic Planning and Execution framework to augment large language models (LLMs) search capabilities.<n>It first employs RL with decomposition-specific rewards to optimize strategic planning.<n>It then applies supervised fine-tuning on high-quality multi-hop trajectories to equip the model with robust iterative sub-task execution capabilities.
  arXiv  Detail & Related papers  (2026-03-14T09:17:12Z)
• SE-Search: Self-Evolving Search Agent via Memory and Dense Reward [87.79131676521656]
  Retrieval augmented generation (RAG) reduces hallucinations and factual errors in large language models (LLMs) by conditioning generation on retrieved external knowledge.<n>Existing methods often accumulate irrelevant or noisy documents and rely on sparse reinforcement learning signals.<n>We propose textbfSelf-textbfEvolving textbfSearch, a Self-Evolving Search agent that improves online search behavior through three components.
  arXiv  Detail & Related papers  (2026-02-06T09:14:07Z)
• Search-R2: Enhancing Search-Integrated Reasoning via Actor-Refiner Collaboration [49.9937230730202]
  We propose Search-R2, a novel Actor-Refiner collaboration framework that enhances reasoning through targeted intervention.<n>Our approach decomposes the generation process into an Actor, which produces initial reasoning trajectories.<n>We show that Search-R2 consistently outperforms strong RAG and RL-based baselines across model scales.
  arXiv  Detail & Related papers  (2026-02-03T15:32:09Z)
• Agentic-R: Learning to Retrieve for Agentic Search [82.43426728778418]
  We propose a novel retriever training framework tailored for agentic search.<n>Unlike retrievers designed for single-turn retrieval-augmented generation (RAG), we propose to use both local query-passage relevance and global answer correctness.<n>Our retriever, termed ours, consistently outperforms strong baselines across different search agents.
  arXiv  Detail & Related papers  (2026-01-17T02:59:54Z)
• Search Self-play: Pushing the Frontier of Agent Capability without Supervision [14.889394507446477]
  Self-play training for deep search agents is proposed in this paper.<n>In this search self-play (SSP) game, the proposer and the solver co-evolve their agent capabilities through both competition and cooperation.<n>SSP can significantly improve search agents' performance uniformly on various benchmarks without any supervision.
  arXiv  Detail & Related papers  (2025-10-21T17:19:35Z)
• Beyond Outcome Reward: Decoupling Search and Answering Improves LLM Agents [19.31471304268234]
  We introduce DeSA (Decoupling Search-and-Answering), a simple two-stage training framework that explicitly separates search optimization from answer generation.<n>Across seven QA benchmarks, DeSA-trained agents consistently improve search behaviors, delivering substantially higher search recall and answer accuracy than outcome-only baselines.
  arXiv  Detail & Related papers  (2025-10-06T11:09:45Z)
• ReSearch: Learning to Reason with Search for LLMs via Reinforcement Learning [74.65632662894086]
  We propose ReSearch, a framework that trains LLMs to Reason with Search via reinforcement learning.<n>Our approach treats search operations as integral components of the reasoning chain, where when and how to perform searches is guided by text-based thinking.<n>Analysis reveals that ReSearch naturally elicits advanced reasoning capabilities such as reflection and self-correction.
  arXiv  Detail & Related papers  (2025-03-25T09:00:58Z)
• Unified Active Retrieval for Retrieval Augmented Generation [69.63003043712696]
  In Retrieval-Augmented Generation (RAG), retrieval is not always helpful and applying it to every instruction is sub-optimal. Existing active retrieval methods face two challenges: 1. They usually rely on a single criterion, which struggles with handling various types of instructions. They depend on specialized and highly differentiated procedures, and thus combining them makes the RAG system more complicated.
  arXiv  Detail & Related papers  (2024-06-18T12:09:02Z)
• ReST-MCTS*: LLM Self-Training via Process Reward Guided Tree Search [50.45155830888697]
  We develop a reinforced self-training approach, called ReST-MCTS*, based on integrating process reward guidance with tree search MCTS* for collecting higher-quality reasoning traces as well as per-step value to train policy and reward models. We first show that the tree-search policy in ReST-MCTS* achieves higher accuracy compared with prior LLM reasoning baselines such as Best-of-N and Tree-of-Thought, within the same search budget.
  arXiv  Detail & Related papers  (2024-06-06T07:40:00Z)
• Zero-Shot Retrieval with Search Agents and Hybrid Environments [8.017306481455778]
  Current language models can learn symbolic query reformulation policies, in combination with traditional term-based retrieval, but fall short of outperforming neural retrievers. We extend the previous learning to search setup to a hybrid environment, which accepts discrete query refinement operations, after a first-pass retrieval step via a dual encoder. Experiments on the BEIR task show that search agents, trained via behavioral cloning, outperform the underlying search system based on a combined dual encoder retriever and cross encoder reranker.
  arXiv  Detail & Related papers  (2022-09-30T13:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.