Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward

• URL: http://arxiv.org/abs/2605.30824v1
• Date: Fri, 29 May 2026 04:18:55 GMT
• Title: Planner-Centric Reinforcement Learning for Deep Research with Structure-Aware Reward
• Abstract summary: DecomposeR is a planner-centric deep research framework that represents research plans as typed directed acyclic graphs (DAGs)<n>We train a Qwen3-8B model in two stages: planner reinforcement learning (RL) first learns graph structure and query decomposition to improve research planning, and answerer reinforcement learning (RL) then learns branch-level execution and final synthesis conditioned on the learned plan.<n> Experiments show that DecomposeR improves over strong comparable open baselines by 5.1-8.0 points on popular long-form benchmarks due to improved planning and answering capabilities.
• Score: 4.833783769369
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep research tasks require LLMs to plan what to investigate, retrieve evidence, and synthesize long-form answers across multiple branches of inquiry. Existing training paradigms either rely on short-form verifiable QA as a proxy or optimize monolithic long trajectories, which makes planning and execution difficult to disentangle and yields weak credit assignment for the planning process. We propose DecomposeR, a planner-centric deep research framework that represents research plans as typed directed acyclic graphs (DAGs), allowing planning to be made explicit, structured, and rewardable. We train a Qwen3-8B model in two stages: planner reinforcement learning (RL) first learns graph structure and query decomposition to improve research planning, and answerer reinforcement learning (RL) then learns branch-level execution and final synthesis conditioned on the learned plan. By assigning rewards to explicit planner tokens and structured components rather than to a flat trajectory, DecomposeR enables finer-grained optimization of planning while reducing the ambiguity of end-to-end training. Experiments show that DecomposeR-8B improves over strong comparable open baselines by 5.1-8.0 points on popular long-form benchmarks due to improved planning and answering capabilities.

Related papers

• PlanningBench: Generating Scalable and Verifiable Planning Data for Evaluating and Training Large Language Models [52.48858778580074]
  Planning is a fundamental capability for large language models (LLMs)<n>PlanningBench is a framework for generating scalable, diverse, verifiable planning data for both evaluation and training.
  arXiv  Detail & Related papers  (2026-05-20T08:10:15Z)
• Distilling Reasoning Without Knowledge: A Framework for Reliable LLMs [0.0]
  We propose a modular framework that separates planning from factual retrieval and answer synthesis.<n>A lightweight student planner is trained via a teacher-student framework to generate structured decompositions.<n>We evaluate the proposed framework on SEAL-0, an extremely challenging benchmark for search-augmented LLMs.
  arXiv  Detail & Related papers  (2026-03-15T16:06:54Z)
• Agentic Planning with Reasoning for Image Styling via Offline RL [66.10749901925941]
  Direct prompt-based editing fails on complex transformations because vague and subjective prompts often require nuanced understanding of what should be changed in the image.<n>We present a tool-based agentic RL post-training framework that addresses this through structured planning with chain-of-thought reasoning.
  arXiv  Detail & Related papers  (2026-03-07T11:14:37Z)
• Plan Then Retrieve: Reinforcement Learning-Guided Complex Reasoning over Knowledge Graphs [52.16166558205338]
  Graph-RFT is a novel two-stage reinforcement fine-tuning KGQA framework with a 'plan-KGsearch-and-Websearch-during-think' paradigm.<n>It enables LLMs to perform autonomous planning and adaptive retrieval scheduling across KG and web sources under incomplete knowledge conditions.
  arXiv  Detail & Related papers  (2025-10-23T16:04:13Z)
• DeepPlanner: Scaling Planning Capability for Deep Research Agents via Advantage Shaping [74.34061104176554]
  We propose DeepPlanner, an end-to-end RL framework that effectively enhances the planning capabilities of deep research agents.<n>Our approach shapes token-level advantage with an entropy-based term to allocate larger updates to high entropy tokens, and selectively upweights sample-level advantages for planning-intensive rollouts.
  arXiv  Detail & Related papers  (2025-10-14T20:47:05Z)
• PLAN-TUNING: Post-Training Language Models to Learn Step-by-Step Planning for Complex Problem Solving [66.42260489147617]
  We introduce PLAN-TUNING, a framework that distills synthetic task decompositions from large-scale language models.<n>Plan-TUNING fine-tunes smaller models via supervised and reinforcement-learning objectives to improve complex reasoning.<n>Our analysis demonstrates how planning trajectories improves complex reasoning capabilities.
  arXiv  Detail & Related papers  (2025-07-10T07:30:44Z)
• Large Language Models for Planning: A Comprehensive and Systematic Survey [29.168219253281453]
  Large Language Models (LLMs) have demonstrated remarkable performance on certain planning tasks.<n>This paper presents a comprehensive review of LLM-based planning.
  arXiv  Detail & Related papers  (2025-05-26T08:44:53Z)
• HyperTree Planning: Enhancing LLM Reasoning via Hierarchical Thinking [109.09735490692202]
  We propose HyperTree Planning (HTP), a novel reasoning paradigm that constructs hypertree-structured planning outlines for effective planning.<n> Experiments demonstrate the effectiveness of HTP, achieving state-of-the-art accuracy on the TravelPlanner benchmark with Gemini-1.5-Pro, resulting in a 3.6 times performance improvement over o1-preview.
  arXiv  Detail & Related papers  (2025-05-05T02:38:58Z)
• DHP: Discrete Hierarchical Planning for Hierarchical Reinforcement Learning Agents [2.1438108757511958]
  We propose a method that replaces continuous distance estimates with discrete reachability checks to evaluate subgoal feasibility.<n>Experiments in 25-room navigation environments demonstrate $100%$ success rate.<n>The method also generalizes to momentum-based control tasks and requires only $log N$ steps for replanning.
  arXiv  Detail & Related papers  (2025-02-04T03:05:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.