DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents

• URL: http://arxiv.org/abs/2606.17029v1
• Date: Mon, 15 Jun 2026 17:52:27 GMT
• Title: DEEPRUBRIC: Evidence-Tree Rubric Supervision for Efficient Reinforcement Learning of Deep Research Agents
• Abstract summary: Reinforcement learning with rubric-based rewards improves deep research agents by optimizing them against checkable criteria that translate report quality into reward signals.<n>Most existing studies ask an LLM to generate rubrics for a given query, but when the model fails to infer the underlying information needs, the generated rubrics may be incomplete and reduce RL efficiency.<n>To obtain more reliable query--rubric supervision, we introduce DeepRubric, a data construction framework that reverses this process.
• Score: 17.420077157633852
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Deep research agents synthesize long-form reports by searching and reasoning over retrieved evidence. Reinforcement learning with rubric-based rewards improves these agents by optimizing them against checkable criteria that translate report quality into reward signals, but its efficiency depends on whether those criteria reliably capture the task scope and evidence needs. Most existing studies ask an LLM to generate rubrics for a given query, but when the model fails to infer the underlying information needs, the generated rubrics may be incomplete and reduce RL efficiency. To obtain more reliable query--rubric supervision, we introduce DeepRubric, a data construction framework that reverses this process: instead of inferring evaluation criteria for a given query, it first determines what an evidence-backed report should be evaluated on and then synthesizes aligned query--rubric pairs from those evaluation targets. Starting from a sampled seed topic, DeepRubric builds an evidence tree by recursively expanding evidence-backed sub-questions, whose leaves serve as atomic and verifiable evaluation targets. It then uses the evidence tree to synthesize the training query and rubrics, ensuring that the reward evaluates exactly the information requested by the query. Using DeepRubric, we construct 9K query--rubric supervision examples and train DeepRubric-8B with rubric-based GRPO, achieving comparable performance to prior open state-of-the-art deep research models across three benchmarks with roughly 13x fewer RL GPU-hours.

Related papers

• Personalized Deep Research Query Refinement with Graph-Scaffolded Evidence Grounding [9.25068659105141]
  G-STEER learns a clarification policy from graph-scaffolded trajectories spanning diverse factor dependencies and evidence conditions.<n>Experiments show that G-STEER achieves the strongest overall weighted target coverage and the highest downstream report personalization across both evaluated DRAs.
  arXiv  Detail & Related papers  (2026-08-06T10:58:17Z)
• QUBRIC: Co-Designing Queries and Rubrics for RL Beyond Verifiable Rewards [68.15832368034815]
  We present a framework that co-designs queries and rubrics.<n>QUBRIC achieves a +5.5 point gain on ArenaHard over the SFT baseline.
  arXiv  Detail & Related papers  (2026-06-02T17:53:04Z)
• Deep Research as Rubric for Reinforcement Learning [35.642522713903794]
  We propose a two-stage framework for constructing open-ended rubrics.<n>DR-rubric-8B supports bootstrap generation without frontier-model assistance.<n>Experiments show that DR-rubric achieves strong competitive performance with only 1K -- 3K training instances.
  arXiv  Detail & Related papers  (2026-05-31T08:25:04Z)
• RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards [76.17893114021757]
  Deep research systems that plan, search, evaluate evidence, and synthesize long-form reports lack ground-truth answers and span many tool-augmented decisions.<n>In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory.<n>We introduceEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution.
  arXiv  Detail & Related papers  (2026-05-11T17:40:38Z)
• Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems [55.04168927059962]
  Reasoning-intensive retrieval aims to surface evidence that supports downstream reasoning rather than merely matching topical similarity.<n>We introduce BRIGHT-Pro, an expert-annotated benchmark that expands each query with multi-aspect gold evidence.<n>We also construct RTriever- Synth, an aspect-decomposed synthetic corpus that generates complementary positives and positive-conditioned hard negatives.
  arXiv  Detail & Related papers  (2026-05-05T17:42:50Z)
• Learning Query-Specific Rubrics from Human Preferences for DeepResearch Report Generation [80.12435680651488]
  We propose a pipeline to train human-preference-aligned query-specific rubric generators tailored for DeepResearch report generation.<n>We first construct a dataset of DeepResearch-style annotated queries with human preferences over paired reports, and train rubric generators via reinforcement learning.<n>We empirically show that our proposed rubric generators deliver more discriminative and better human-aligned supervision than existing rubric design strategies.
  arXiv  Detail & Related papers  (2026-02-03T15:09:56Z)
• Chaining the Evidence: Robust Reinforcement Learning for Deep Search Agents with Citation-Aware Rubric Rewards [60.0970117192627]
  Reinforcement learning (RL) has emerged as a critical technique for enhancing LLM-based deep search agents.<n>Existing approaches primarily rely on binary outcome rewards, which fail to capture the comprehensiveness and factuality of agents' reasoning process.<n>We propose textbfCitation-aware RL Rewards (CaRR), a fine-grained reward framework for deep search agents.
  arXiv  Detail & Related papers  (2026-01-09T18:57:53Z)
• BrowseComp-Plus: A More Fair and Transparent Evaluation Benchmark of Deep-Research Agent [74.10138164281618]
  BrowseComp-Plus is a benchmark derived from BrowseComp, employing a fixed, carefully curated corpus.<n>This benchmark allows comprehensive evaluation and disentangled analysis of deep research agents and retrieval methods.
  arXiv  Detail & Related papers  (2025-08-08T17:55:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.