Plan Before Search: Search Agents Need Plan

• URL: http://arxiv.org/abs/2605.28354v1
• Date: Wed, 27 May 2026 11:54:15 GMT
• Title: Plan Before Search: Search Agents Need Plan
• Abstract summary: We study the dependency structure among sub-skills and the possibility that distillation is not the only route to capability acquisition.<n>We propose a self-bootstrapping paradigm in which a small-scale seed model generates filtered trajectories that activate Plan in any target model.
• Score: 66.42962362539934
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Training large language models as retrieval-augmented reasoning agents typically combines reinforcement learning with an SFT cold start distilled from a stronger model. However, this paradigm overlooks two fundamental factors: the dependency structure among sub-skills, and the possibility that distillation is not the only route to capability acquisition. We study this through Plan, a structured agentic behavior for multi-hop retrieval that decomposes a question into ordered sub-questions before any retrieval is performed, so that each search step can be anchored to a pre-designed sub-question instead of drifting under the influence of partially relevant documents retrieved earlier. However, across three model families spanning 3B to 14B parameters, we find that an identical reward signal induces qualitatively different RL failure modes. This phenomenon indicates that successful training hinges not only on reward design but also on model-specific feasibility conditions: sufficient initial entropy, training stability, and prerequisite sub-skills. Motivated by this, we propose a self-bootstrapping paradigm in which a small-scale seed model generates filtered trajectories that activate Plan in any target model, eliminating the need for distillation from an external stronger model. Our pipeline activates Plan across every tested model and consistently outperforms competitive baselines on multi-hop QA benchmarks.

Related papers

• DiffusionOPD: A Unified Perspective of On-Policy Distillation in Diffusion Models [55.01951088768769]
  DiffusionOPD is a new multi-task training paradigm for diffusion models based on Online Policy Distillation (OPD)<n>We show that DiffusionOPD consistently surpasses both multi-reward RL and cascade RL baselines in training efficiency and final performance.
  arXiv  Detail & Related papers  (2026-05-14T16:49:09Z)
• On Predicting the Post-training Potential of Pre-trained LLMs [60.07459271263409]
  We introduce a new task of predicting post-training potential - forecasting a base model's performance before post-training.<n>We propose RuDE, a unified framework that bypasses the generation gap of base models by leveraging response discrimination.<n>Experiments demonstrate a correlation greater than 90% with post-training performance.
  arXiv  Detail & Related papers  (2026-05-12T11:33:49Z)
• Regularized Latent Dynamics Prediction is a Strong Baseline For Behavioral Foundation Models [35.088440282359024]
  Behavioral Foundation Models (BFMs) produce agents with the capability to adapt to any unknown reward or task.<n>These methods are only able to produce near-optimal policies for the reward functions that are in the span of some pre-existing state features.<n>We propose an approach, Regularized Latent Dynamics Prediction (RLDP), that can match or surpass state-of-the-art complex representation learning methods for zero-shot RL.
  arXiv  Detail & Related papers  (2026-03-16T19:39:27Z)
• Answer First, Reason Later: Aligning Search Relevance via Mode-Balanced Reinforcement Learning [7.006180736433431]
  Building a search relevance model that achieves both low latency and high performance is a long-standing challenge in the search industry.<n>We propose a novel textbfAnswer-First, Reason Later (AFRL) paradigm.<n>This paradigm requires the model to output the definitive relevance score in the very first token, followed by a structured logical explanation.
  arXiv  Detail & Related papers  (2026-02-10T17:28:12Z)
• Reasoning Palette: Modulating Reasoning via Latent Contextualization for Controllable Exploration for (V)LMs [49.66344956133349]
  Reasoning capacity shapes both inference-time performance and reinforcement learning (RL) training for large (vision-) language models.<n>This paper proposes Reasoning Palette, a novel latent-modulation framework that endows the model with a latent variable for strategic contextualization.
  arXiv  Detail & Related papers  (2025-12-19T03:32:53Z)
• Reasoning with Sampling: Your Base Model is Smarter Than You Think [52.639108524651846]
  We propose a simple iterative sampling algorithm leveraging the base models' own likelihoods.<n>We show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL.<n>Our method does not require training, curated datasets, or a verifier.
  arXiv  Detail & Related papers  (2025-10-16T17:18:11Z)
• DecoupleSearch: Decouple Planning and Search via Hierarchical Reward Modeling [56.45844907505722]
  We propose DecoupleSearch, a framework that decouples planning and search processes using dual value models.<n>Our approach constructs a reasoning tree, where each node represents planning and search steps.<n>During inference, Hierarchical Beam Search iteratively refines planning and search candidates with dual value models.
  arXiv  Detail & Related papers  (2025-09-07T13:45:09Z)
• Metis-RISE: RL Incentivizes and SFT Enhances Multimodal Reasoning Model Learning [20.515599491717442]
  We introduce textbfMetis-RISE (textbfRL textbfSFT textbfEnhances) for multimodal reasoning model learning.
  arXiv  Detail & Related papers  (2025-06-16T02:56:13Z)
• S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models [2.9925837108958864]
  Test-Time Scaling emerges as an active research focus in the large language model community.<n>Recent studies reveal that reasoning models (even Qwen3) consistently exhibit excessive thought redundancy.<n>This paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm.
  arXiv  Detail & Related papers  (2025-05-12T15:50:44Z)
• Unified Multimodal Chain-of-Thought Reward Model through Reinforcement Fine-Tuning [31.727984223052648]
  This paper proposes UnifiedReward-Think, the first unified multimodal CoT-based reward model.<n>We first use a small amount of image generation preference data to distill the reasoning process of GPT-4o.<n>We then prepare large-scale unified multimodal preference data to elicit the model's reasoning process across various vision tasks.
  arXiv  Detail & Related papers  (2025-05-06T08:46:41Z)
• Provable Meta-Learning with Low-Rank Adaptations [37.120226706944926]
  We introduce a framework for generic PEFT-based meta-learning to learn a model that can easily adapt to unseen tasks.<n>For linear models using LoRA, we show that standard retraining is provably suboptimal for finding an adaptable set of parameters.<n>We verify these theoretical insights through experiments on synthetic data as well as real-data vision and language tasks.
  arXiv  Detail & Related papers  (2024-10-29T17:24:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.