Counterfactual Shapley Credit Assignment

• URL: http://arxiv.org/abs/2607.16999v1
• Date: Sat, 18 Jul 2026 23:28:20 GMT
• Title: Counterfactual Shapley Credit Assignment
• Abstract summary: Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents.<n>We introduce Counterfactual Shapley Credit Assignment, a novel framework that attributes credit and blame via the Counterfactual Shapley ($$-value)<n>We derive a consistent estimator that computes $$$-values, enabling a new class of policy methods, $$-PPO, combined with Prioritized Trajectory Replay (PTR)
• Score: 49.39649767581182
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: The Credit Assignment Problem (CAP) is fundamental to developing efficient and explainable Reinforcement Learning (RL) agents. Existing frameworks, whether relying on temporal contiguity or hindsight-conditioned reward reweighting, frequently fail to attribute properly between an agent's policy (skill) and environmental stochasticity (luck). A principled approach to CAP must isolate the true causal drivers of observed outcomes from spurious correlations and environmental randomness. We introduce Counterfactual Shapley Credit Assignment, a novel framework grounded in causal theory that attributes credit and blame via the Counterfactual Shapley Value ($φ$-value). By redistributing environmental rewards, $φ$-values enhance temporal credit assignment across three critical dimensions: sparse causality, high stochasticity, and delayed rewards, all while preserving the optimal policy. We derive a consistent estimator that computes $φ$-values efficiently, enabling a new class of policy gradient methods, $φ$-PPO, combined with Prioritized Trajectory Replay (PTR). Empirical results demonstrate that $φ$-values align precisely to the ground truth causes of task rewards with superior sample efficiency in challenging environments where prior state-of-the-art methods fail to converge.

Related papers

• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• TRACE: A Unified Rollout Budget Allocation Framework for Efficient Agentic Reinforcement Learning [52.67756371231985]
  Reinforcement learning with verifiable rewards (RLVR) is a promising approach for enhancing reasoning and agentic behavior in large language models.<n>We introduce Tree Rollout Allocation for Contrastive Exploration (TRACE), a unified rollout allocation framework.<n>Technically, TRACE allocates rollout budget to both prompt roots and intermediate prefixes that are most likely to yield mixed terminal rewards.
  arXiv  Detail & Related papers  (2026-06-09T17:16:03Z)
• Learning When to Act: Communication-Efficient Reinforcement Learning via Run-Time Assurance [0.27998963147546146]
  We show that a single policy can learn control inputs and communication-efficient timing decisions under a pointwise Lyapunov safety shield.<n>A run-time assurance layer overrides the policy via a one-step-ahead Lyapunov prediction and a precomputed LQR backup.<n>A 12-state 3D quadrotor case study extends the framework to higher-dimensional systems where classical STC is intractable.
  arXiv  Detail & Related papers  (2026-05-11T23:55:15Z)
• Your Language Model is Its Own Critic: Reinforcement Learning with Value Estimation from Actor's Internal States [14.658333957936769]
  We introduce Policy Optimization with Internal State Value Estimation.<n>It obtains a baseline at negligible cost by using the policy model's internal signals already computed during the policy forward pass.<n>A lightweight probe predicts the expected verifiable reward from the hidden states of the prompt and generated trajectory.
  arXiv  Detail & Related papers  (2026-05-08T10:49:36Z)
• Utilizing and Calibrating Hindsight Process Rewards via Reinforcement with Mutual Information Self-Evaluation [50.696688705287755]
  We propose Mutual Information Self-Evaluation to overcome the sparse reward challenge in reinforcement learning.<n>MISE enables an agent to learn autonomously from dense internal rewards supplementing sparse extrinsic signals.<n>We prove that utilizing hindsight self-evaluation rewards is equivalent to minimizing an objective that combines mutual information with a KL divergence term between the policy and a proxy reward policy.
  arXiv  Detail & Related papers  (2026-04-13T15:18:51Z)
• $V_{0.5}$: Generalist Value Model as a Prior for Sparse RL Rollouts [81.48669089692189]
  Generalist Value Models (such as $V_0.5$) achieve pre-trained value estimation by explicitly encoding model capabilities in-context.<n>We propose $V_0.5$, which adaptively fuses the baseline predicted by such value model with the empirical mean derived from sparse rollouts.<n>$V_0.5$ significantly outperforms GRPO and DAPO, achieving faster convergence and over some 10% performance improvement.
  arXiv  Detail & Related papers  (2026-03-11T14:57:41Z)
• Robust Regularized Policy Iteration under Transition Uncertainty [6.7431287237221085]
  We formulate offline RL as robust policy optimization, treating the transition kernel as a decision variable within an uncertainty set.<n>We propose Robust Regularized Policy Iteration (RRPI), which replaces the intractable max-min bilevel objective with a tractable KL-regularized surrogate.<n>Experiments on D4RL benchmarks demonstrate that RRPI achieves strong average performance, outperforming recent baselines.
  arXiv  Detail & Related papers  (2026-03-10T08:18:27Z)
• On Corruption-Robustness in Performative Reinforcement Learning [13.509499718691016]
  We study the convergence of repeated retraining approaches to a performatively stable policy.<n>We extend these approaches to operate under corrupted data.<n>We prove that our approach exhibits last-ite convergence to an approximately stable policy.
  arXiv  Detail & Related papers  (2025-05-08T19:37:35Z)
• Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning [14.852334980733369]
  Credit assignmen, disentangling each agent's contribution to a shared reward, is a critical challenge in cooperative multi-agent reinforcement learning.<n>We introduce Temporal-Agent Reward Redistribution (TAR$2$), an approach that decouples credit modeling from this constraint.<n>We demonstrate that this method is equivalent to a valid Potential-Based Reward Shaping (PBRS), which guarantees the optimal policy is preserved regardless of model accuracy.
  arXiv  Detail & Related papers  (2025-02-07T12:07:57Z)
• Distributionally Robust Policy Learning under Concept Drifts [33.44768994272614]
  This paper studies a more nuanced problem -- robust policy learning under the concept drift.<n>We first provide a doubly-robust estimator for evaluating the worst-case average reward of a given policy.<n>We then propose a learning algorithm that outputs the policy maximizing the estimated policy value within a given policy class.
  arXiv  Detail & Related papers  (2024-12-18T19:53:56Z)
• Would I have gotten that reward? Long-term credit assignment by counterfactual contribution analysis [50.926791529605396]
  We introduce Counterfactual Contribution Analysis (COCOA), a new family of model-based credit assignment algorithms. Our algorithms achieve precise credit assignment by measuring the contribution of actions upon obtaining subsequent rewards.
  arXiv  Detail & Related papers  (2023-06-29T09:27:27Z)
• STAS: Spatial-Temporal Return Decomposition for Multi-agent Reinforcement Learning [10.102447181869005]
  We introduce a novel method that learns credit assignment in both temporal and spatial dimensions. Our results demonstrate that our method effectively assigns spatial-temporal credit, outperforming all state-of-the-art baselines.
  arXiv  Detail & Related papers  (2023-04-15T10:09:03Z)
• False Correlation Reduction for Offline Reinforcement Learning [115.11954432080749]
  We propose falSe COrrelation REduction (SCORE) for offline RL, a practically effective and theoretically provable algorithm. We empirically show that SCORE achieves the SoTA performance with 3.1x acceleration on various tasks in a standard benchmark (D4RL)
  arXiv  Detail & Related papers  (2021-10-24T15:34:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.