RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards

• URL: http://arxiv.org/abs/2605.10899v1
• Date: Mon, 11 May 2026 17:40:38 GMT
• Title: RubricEM: Meta-RL with Rubric-guided Policy Decomposition beyond Verifiable Rewards
• Abstract summary: Deep research systems that plan, search, evaluate evidence, and synthesize long-form reports lack ground-truth answers and span many tool-augmented decisions.<n>In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory.<n>We introduceEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution.
• Score: 76.17893114021757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training deep research agents, namely systems that plan, search, evaluate evidence, and synthesize long-form reports, pushes reinforcement learning beyond the regime of verifiable rewards. Their outputs lack ground-truth answers, their trajectories span many tool-augmented decisions, and standard post-training offers little mechanism for turning past attempts into reusable experience. In this work, we argue that rubrics should serve not merely as final-answer evaluators, but as the shared interface that structures policy execution, judge feedback, and agent memory. Based on this view, we introduce RubricEM, a rubric-guided reinforcement learning framework that combines stagewise policy decomposition with reflection-based meta-policy evolution. RubricEM first makes research trajectories stage-aware by conditioning planning, evidence gathering, review, and synthesis on self-generated rubrics. It then assigns credit with Stage-Structured GRPO, which uses stagewise rubric judgments to provide denser semantic feedback for long-horizon optimization. In parallel, RubricEM trains a shared-backbone reflection meta-policy that distills judged trajectories into reusable rubric-grounded guidance for future attempts. The resulting RubricEM-8B achieves strong performance across four long-form research benchmarks, outperforming comparable open models and approaching proprietary deep-research systems. Beyond final performance, we perform thorough analyses to understand the key ingredients of RubricEM.

Related papers

• EDGE: Experience-Distillation for Guided Exploration in Agentic Reinforcement Learning [25.200068256200293]
  reusable exploration patterns embedded in interaction trajectories are largely discarded after a single policy update.<n>We propose EDGE (Experience-Distillation for Guided Exploration), a framework that treats retrieved experiences as temporary training-time scaffolds.<n> EDGE improves over strong RL baselines by up to 12.5 points and remains effective without inference-time scaffolds or a proprietary reflector.
  arXiv  Detail & Related papers  (2026-08-22T13:16:25Z)
• From Holistic Evaluation to Structured Criteria: Rubrics Across the Evolving LLM Landscape [79.30826980815927]
  rubrics are explicit criteria sets that transform complex quality judgments into structured and actionable standards.<n>We systematically organize existing rubric designs, examine their construction and optimization, and analyze their role across evaluation and training.
  arXiv  Detail & Related papers  (2026-06-07T13:34:55Z)
• Beyond Classification: Dynamic Adapter Routing for Continual Multimodal Retrieval [16.629210107453464]
  We introduce a principled evaluation framework for continual multimodal retrieval (CMR) spanning diverse visual domains.<n>Our empirical analysis shows that standard CIL methods fail to yield meaningful gains in our more challenging scenario.<n>We propose Dynamic Routing Adapter (DAR), a novel approach based on adapters selected through prototype-based routing and combined via model merging.
  arXiv  Detail & Related papers  (2026-05-29T12:32:59Z)
• Co-ReAct: Rubrics as Step-Level Collaborators for ReAct Agents [6.89645931986174]
  We introduce Co-ReAct, a rubric-guided action-selection framework.<n>At each decision step, Co-ReAct injects a rubric into the agent's context to guide the next Reason-or-Act decision.<n>We train a dedicated rubric generator with GRPO to make this guidance reliable.
  arXiv  Detail & Related papers  (2026-05-22T12:59:16Z)
• Rethinking Reasoning-Intensive Retrieval: Evaluating and Advancing Retrievers in Agentic Search Systems [55.04168927059962]
  Reasoning-intensive retrieval aims to surface evidence that supports downstream reasoning rather than merely matching topical similarity.<n>We introduce BRIGHT-Pro, an expert-annotated benchmark that expands each query with multi-aspect gold evidence.<n>We also construct RTriever- Synth, an aspect-decomposed synthetic corpus that generates complementary positives and positive-conditioned hard negatives.
  arXiv  Detail & Related papers  (2026-05-05T17:42:50Z)
• CoSearch: Joint Training of Reasoning and Document Ranking via Reinforcement Learning for Agentic Search [51.911048955965136]
  CoSearch is a framework that jointly trains a multi-step reasoning agent and a generative document ranking model.<n>Our results show that joint training of the reasoning agent and retrieval system is both feasible and strongly performant.
  arXiv  Detail & Related papers  (2026-04-19T17:48:17Z)
• Chained Prompting for Better Systematic Review Search Strategies [0.6633201258809686]
  We introduce a Large Language Model-based chained prompt engineering framework for the automated development of search strategies in systematic reviews.<n>The framework replicates the procedural structure of manual search design while leveraging LLMs to decompose review objectives, extract and PICO elements, generate conceptual representations, expand terminologies, and synthesize queries.
  arXiv  Detail & Related papers  (2025-11-28T12:12:38Z)
• Thinking Forward and Backward: Multi-Objective Reinforcement Learning for Retrieval-Augmented Reasoning [137.33138614095435]
  Retrieval-augmented generation (RAG) has proven to be effective in mitigating hallucinations in large language models.<n>Recent efforts have incorporated search-based interactions into RAG, enabling iterative reasoning with real-time retrieval.<n>We propose Bi-RAR, a novel retrieval-augmented reasoning framework that evaluates each intermediate step jointly in both forward and backward directions.
  arXiv  Detail & Related papers  (2025-11-12T08:29:39Z)
• Tree-OPO: Off-policy Monte Carlo Tree-Guided Advantage Optimization for Multistep Reasoning [3.6333725470852443]
  We explore how Monte Carlo Tree Search can be repurposed to improve policy optimization in preference-based reinforcement learning.<n>We propose a staged GRPO training paradigm where completions are derived from partially revealed MCTS rollouts, introducing a novel tree-structured setting for advantage estimation.<n>Our initial results indicate that while structured advantage estimation can stabilize and better reflect reasoning quality, challenges such as advantage saturation and reward signal collapse remain.
  arXiv  Detail & Related papers  (2025-09-11T09:18:07Z)
• STARec: An Efficient Agent Framework for Recommender Systems via Autonomous Deliberate Reasoning [54.28691219536054]
  We introduce STARec, a slow-thinking augmented agent framework that endows recommender systems with autonomous deliberative reasoning capabilities.<n>We develop anchored reinforcement training - a two-stage paradigm combining structured knowledge distillation from advanced reasoning models with preference-aligned reward shaping.<n>Experiments on MovieLens 1M and Amazon CDs benchmarks demonstrate that STARec achieves substantial performance gains compared with state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-08-26T08:47:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.