ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents

• URL: http://arxiv.org/abs/2606.21262v1
• Date: Fri, 19 Jun 2026 09:38:02 GMT
• Title: ARCO: Adaptive Rubric with Co-Evolution for Multi-Step LLM-Based Agents
• Abstract summary: ARCO is an adaptive rubric framework for reinforcement learning.<n>It improves the best EM in every setting over strong outcome-, rubric-, and process-reward baselines.<n>Its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior.
• Score: 12.091072999307622
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning for multi-step LLM agents often relies on scalar rewards that indicate success but cannot explain why a trajectory is good or bad. Rubric-based rewards improve interpretability through natural-language criteria, but existing methods score at the trajectory level and freeze the scorer behind a closed-source judge, leaving step-level credit assignment unresolved and the judge itself static. We propose ARCO (Adaptive Rubric CO-evolution), a rubric framework in which a same-scale model $μ$ shares a backbone with two heads: a generation head that produces per-step criteria, and a score head that predicts rubric-conditioned step-level rewards. A trajectory decomposition constraint ties the sum of step rewards to the terminal outcome, enabling credit assignment without step-level labels, while $μ$ and the policy $π$ are jointly updated on on-policy data so that the rubric content and the scoring function co-evolve at the parameter level. Across HotpotQA, 2WikiMultiHopQA, and MuSiQue with two open-source backbones, ARCO improves the best EM in every setting over strong outcome-, rubric-, and process-reward baselines, and analyses show that its rubrics are step-specific, robust to design choices, and useful for diagnosing agent behavior. Codes and data are available at https://github.com/zihangtian/ARCO.

Related papers

• Contrastive Branch Policy Optimization [16.16893757028268]
  Reinforcement learning with verifiable rewards (RLVR) enables language models to learn multi-turn interaction with external tools.<n>Existing methods tend to conflate two distinct problems: allocating a fixed rollout budget and translating branch outcomes into token-level credit.<n>We introduce Contrastive Branch Policy Optimization (CBPO), which disentangles these two problems and assigns a dedicated mechanism to each.
  arXiv  Detail & Related papers  (2026-08-25T09:25:55Z)
• AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
  AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
  arXiv  Detail & Related papers  (2026-08-06T13:00:59Z)
• When Does Trajectory-Level Supervision Permit Efficient Offline Reinforcement Learning? [10.71573326176278]
  We develop a statistical theory for offline policy optimization from outcomelevel supervision.<n>OPAC is a pessimistic actor algorithm that learns a reward model and optimize a policy from trajectory-level labels.<n>We study generalized outcome-based offline RL, where both the supervision and the objective are trajectory-level quantities induced by a nonlinear aggregation of latent per-step rewards.
  arXiv  Detail & Related papers  (2026-06-16T22:55:45Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• ARBOR: Online Process Rewards via a Reusable Rubric Buffer for Search Agents [48.80766702702854]
  LLM-based search agents are trained predominantly with outcome-only reward, leaving the search process itself unsupervised.<n>This signal degenerates on outcome-homogeneous groups where all sampled trajectories share the same correctness, yielding zero within-group advantage and no gradient.<n>We propose ARBOR (Adaptive Buffer for Online Reward), a reusable process-reward framework that maintains a rubric memory shared across queries.
  arXiv  Detail & Related papers  (2026-06-02T06:58:54Z)
• StepOPSD: Step-Aware Online Preference Distillation for Agent Reinforcement Learning [6.365332042924078]
  We present StepOPSD, a post-rollout preference self-distillation framework that takes the agent step as the unit of credit redistribution.<n>StepOPSD decomposes trajectories into action-centered step segments, rescoring them under hindsight-enriched teacher contexts.<n>Results reveal a consistent two-knob law: smaller _clip acts as a broadly stabilizing local trust region, whereas the optimal global mixing strength _mix remains task-dependent.
  arXiv  Detail & Related papers  (2026-05-26T15:07:03Z)
• Verifier-Free RL for LLMs via Intrinsic Gradient-Norm Reward [69.99652051809737]
  In this work, we propose Verifier-free Intrinsic Gradient-Norm Reward (VIGOR)<n>VIGOR is a simple reward that uses only the policy model itself.<n>It exhibits cross-domain transfer to code benchmarks when trained only on math data.
  arXiv  Detail & Related papers  (2026-05-11T03:15:37Z)
• Rubrics to Tokens: Bridging Response-level Rubrics and Token-level Rewards in Instruction Following Tasks [30.108876878365496]
  Reinforcement Learning (RL) has emerged as a promising approach for aligning Large Language Models with complex, open-domain instruction following tasks.<n>Existing methods predominantly rely on response-level rewards, introducing severe reward sparsity and reward ambiguity problems.<n>We propose a novel rubric-based RL framework that bridges coarse response-level scores and fine-grained token-level credit assignment.
  arXiv  Detail & Related papers  (2026-04-03T07:02:57Z)
• SQL-ASTRA: Alleviating Sparse Feedback in Agentic SQL via Column-Set Matching and Trajectory Aggregation [51.696027315573296]
  We propose a framework featuring a universal two-tiered reward mechanism designed to provide effective trajectory-level evaluation and dense step-level signals.<n>In traditional paradigms, rewards are determined solely by the final-turn feedback, which ignores the intermediate process and leads to ambiguous credit evaluation.<n>We show that our approach outperforms SOTA Arctic-Text2-R1-7B on BIRD 2.0 using identical models, propelling Text-to-reward convergence toward a robust multi-turn agent paradigm.
  arXiv  Detail & Related papers  (2026-03-17T06:26:36Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Discriminative Policy Optimization for Token-Level Reward Models [55.98642069903191]
  Process reward models (PRMs) provide more nuanced supervision compared to outcome reward models (ORMs)<n>Q-RM explicitly learns token-level Q-functions from preference data without relying on fine-grained annotations.<n>Reinforcement learning with Q-RM significantly enhances training efficiency, achieving convergence 12 times faster than ORM on GSM8K and 11 times faster than step-level PRM on MATH.
  arXiv  Detail & Related papers  (2025-05-29T11:40:34Z)
• Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning [14.852334980733369]
  Credit assignmen, disentangling each agent's contribution to a shared reward, is a critical challenge in cooperative multi-agent reinforcement learning.<n>We introduce Temporal-Agent Reward Redistribution (TAR$2$), an approach that decouples credit modeling from this constraint.<n>We demonstrate that this method is equivalent to a valid Potential-Based Reward Shaping (PBRS), which guarantees the optimal policy is preserved regardless of model accuracy.
  arXiv  Detail & Related papers  (2025-02-07T12:07:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.