TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning

• URL: http://arxiv.org/abs/2606.32017v2
• Date: Mon, 06 Jul 2026 17:49:31 GMT
• Title: TRIAGE: Role-Typed Credit Assignment for Agentic Reinforcement Learning
• Abstract summary: Agentic reinforcement learning requires assigning credit to environment-facing actions.<n>Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens.<n>We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit.
• Score: 10.917273110894469
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Agentic reinforcement learning requires assigning credit to environment-facing actions such as searches, clicks, edits, navigation commands, and object interactions. Standard GRPO uses the final verifier outcome as a uniform advantage over all action tokens. This outcome signal is useful but structurally incomplete: it punishes useful exploration in failed rollouts and reinforces redundant or regressive actions in successful rollouts. We propose TRIAGE, a role-typed credit assignment framework that adds a semantic role axis to outcome credit. A structured judge classifies each segment as decisive progress, useful exploration, no-progress infrastructure, or regression, and a fixed role-conditioned rule maps these labels to bounded segment-level process rewards. This keeps verifier outcomes as the source of optimization direction while correcting the two main blind spots of outcome-only credit. We further show that role-conditioned credit is the optimal segment-level correction expressible from role labels alone -- a projection of the per-segment advantage residual onto the role variable -- so that the fixed role constants reduce advantage estimation error whenever the judge is reliable, and we connect this to lower-variance policy gradients. Across ALFWorld, Search-QA, and WebShop, TRIAGE improves success rates over GRPO for two policy models and outperforms both a scalar judge-derived process reward and an outcome-supervised shared-backbone value baseline. Ablations show that the gain comes from role typing rather than merely adding dense rewards: reliable detection of regression inside successful trajectories is the dominant contributor, while exploration credit provides a consistent secondary gain; on completed ALFWorld and WebShop rollouts, TRIAGE also reduces environment-facing turns by an additional $10.4\%$ and $14.8\%$ relative to GRPO.

Related papers

• Reconciling Process Supervision with Outcome-Based Credit in Agentic Policy Optimization [13.354353071601379]
  We introduce TASPO, which converts privileged supervision into outcome-grounded action credit.<n>Across three agentic benchmarks, TASPO improves over GRPO by 10.6% and generalizes better to unseen tasks.
  arXiv  Detail & Related papers  (2026-08-31T16:51:50Z)
• AgentOPSD: Recursive Self-Distillation for Agentic Reinforcement Learning [58.76655851910778]
  AgentOPSD is a critic-free, recursive method for turn-level credit assignment in agentic reinforcement learning.<n>We evaluate AgentOPSD on ALFWorld, WebShop, and Search-QA using Qwen2.5 models at two scales (3B and 7B)
  arXiv  Detail & Related papers  (2026-08-06T13:00:59Z)
• STAMP: Provenance-Guided Credit Assignment for Deep Search Agents [16.1299556246389]
  Reinforcement learning for deep-search agents has largely focused on trajectory-level scoring.<n>We propose STAMP, in which a reference-based verifier judges whether each cited document supports an entity or relation.<n>On BrowseComp, BrowseComp-ZH, and xbench-DS, STAMP improves the GRPO baseline by +2.0/+5.5/+3.0 points.
  arXiv  Detail & Related papers  (2026-07-13T07:12:55Z)
• HERO: Hindsight-Enhanced Reflection from Environment Observations for Agentic Self-Distillation [50.53459634301361]
  HERO is a hindsight-enhanced self-distillation framework that uses next environment observations as locally aligned feedback.<n> HERO improves task success and reduces unnecessary turns over environment-feedback-only self-distillation and GRPO.
  arXiv  Detail & Related papers  (2026-06-10T01:35:34Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• PBSD: Privileged Bayesian Self-Distillation for Long-Horizon Credit Assignment [35.978805768172656]
  Long-horizon agentic tasks pose a fundamental credit assignment challenge for outcome-base reinforcement learning.<n>We propose PBSD (Privileged Bayesian Self-Distillation), a Bayes-calibrated self-distillation method for fine-grained credit assignment under sparse final rewards.
  arXiv  Detail & Related papers  (2026-06-08T11:20:58Z)
• TAPO: Tool-Aware Policy Optimization via Credit Transfer for Multimodal Search Agents [54.08846865906602]
  We characterize credit misassignment as a systematic failure mode of GRPO in tool-augmented multimodal search agents.<n>We propose Tool-Aware Policy Optimization (TAPO), which exploits the parameter-determinism property of information-acquisition tools.
  arXiv  Detail & Related papers  (2026-06-04T07:15:43Z)
• VeriGate: Verifier-Gated Step-Level Supervision for GRPO [51.26100506256885]
  Group Relative Policy Optimization is an effective recipe for training reasoning models with verifier-based outcome rewards.<n>We present VeriGate, a verifier-gated extension of GRPO that addresses these limitations with three design choices.<n>We show that VeriGate improves average accuracy by about 20% and 12% for 1.5B and 7B models respectively.
  arXiv  Detail & Related papers  (2026-05-28T18:20:32Z)
• Outcome-Grounded Advantage Reshaping for Fine-Grained Credit Assignment in Mathematical Reasoning [60.00161035836637]
  Group Relative Policy Optimization has emerged as a promising critic-free reinforcement learning paradigm for reasoning tasks.<n>We introduce Outcome-grounded Advantage Reshaping (OAR), a fine-grained credit assignment mechanism that redistributes advantages based on how much each token influences the model's final answer.<n>OAR-G achieves comparable gains with negligible computational overhead, both significantly outperforming a strong GRPO baseline.
  arXiv  Detail & Related papers  (2026-01-12T10:48:02Z)
• Pinpointing crucial steps: Attribution-based Credit Assignment for Verifiable Reinforcement Learning [5.880405013005892]
  ACPO is a phased framework that incorporates a difficulty-aware curriculum.<n>ACPO improves exploration by using trajectory semantic segmentation and an attribution-based representation.<n>It enhances exploitation with a factorized reward system that precisely quantifies the hierarchical contribution of each reasoning step.
  arXiv  Detail & Related papers  (2025-10-10T01:22:55Z)
• Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis [50.926791529605396]
  We introduce Counterfactual Contribution Analysis (COCOA), a new family of model-based credit assignment algorithms. Our algorithms achieve precise credit assignment by measuring the contribution of actions upon obtaining subsequent rewards.
  arXiv  Detail & Related papers  (2023-06-29T09:27:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.