RREDCoT: Segment-Level Reward Redistribution for Reasoning Models

• URL: http://arxiv.org/abs/2606.06475v1
• Date: Thu, 04 Jun 2026 17:56:31 GMT
• Title: RREDCoT: Segment-Level Reward Redistribution for Reasoning Models
• Abstract summary: We introduce RREDCoT (Reward REDistribution for Chain of Thoughts) to approximate the optimal reward redistribution without additional generation.<n>We also analyze several aspects relevant to the construction of the redistribution such as segmentation of CoT traces and state value estimation.
• Score: 20.726685669562496
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in reasoning language models have been driven by Reinforcement Learning (RL) fine-tuning. Most often, these rely on the Group Relative Policy Optimization (GRPO) algorithm or modifications thereof to steer the models to produce Chain-of-Thought (CoT) traces. The final answer can only be verified, and the reward assigned, after the CoT trace is complete, making it a delayed reward problem. GRPO and its modifications correspond to Monte Carlo methods in standard RL, which are known to suffer from high variance. A possible solution to this problem is the redistribution of rewards through credit assignment, where segments of the CoT trace that are important for arriving at the desirable solution are emphasized by assigning a higher reward. While Monte Carlo sampling can be used to provide an unbiased estimate of intermediate state values, its computational overhead makes it unsuitable for train-time credit assignment in long contexts at high granularity. We introduce RREDCoT (Reward REDistribution for Chain of Thoughts), which utilizes the model itself to approximate the optimal reward redistribution without additional generation. We investigate the advantages of our method compared to MC sampling and several attribution methods. We further analyze several aspects relevant to the construction of the redistribution such as segmentation of CoT traces and state value estimation.

Related papers

• RRC: Unlocking Generative Reward Models in LLM Reinforcement Learning via Ranking-Based Reward Construction [44.617234554495276]
  We propose a Ranking-based Reward Construction (RRC) approach to provide more effective reinforcement learning signals.<n>Our experiments demonstrate that RRC substantially improves RL training with generative reward models, achieving consistent gains over existing reward construction approaches.
  arXiv  Detail & Related papers  (2026-08-06T17:24:36Z)
• When Correct Solutions Repeat: Rarity-Aware Credit Redistribution for GRPO [25.979415525596007]
  Reinforcement learning with verifiable rewards (RLVR)<n>We formalize this behavior as multiplicity-induced structure-level credit concentration.<n> Cue-GRPO instantiates this rule with- out auxiliary-model inference.
  arXiv  Detail & Related papers  (2026-08-04T11:02:26Z)
• Optimizing Visual Generative Models via Distribution-wise Rewards [57.46109819595665]
  We present a novel framework that finetunes generative models using distribution-wise rewards, ensuring better alignment with real-world data distributions.<n>Our approach significantly improves FID-50K across various base models, from 5.77 for SiT and from 3.74 to 3.52 for EDM2.
  arXiv  Detail & Related papers  (2026-07-02T15:08:56Z)
• Coupled Variational Reinforcement Learning for Language Model General Reasoning [83.82392089177841]
  We propose textitbCoupled bVari bReinforcement bLearning (CoVRL) to bridge variational inference and reinforcement learning.<n>CoVRL improves performance by 12.4% over the base model and achieves an additional 2.3% improvement over strong state-of-the-art verifier-free RL baselines.
  arXiv  Detail & Related papers  (2025-12-14T07:03:51Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• Rewarding the Unlikely: Lifting GRPO Beyond Distribution Sharpening [36.81125165911328]
  Reinforcement learning is emerging as a primary driver for improving language model reasoning capabilities.<n>We investigate whether current reinforcement learning algorithms merely sharpen the base model's distribution around problems it can already solve.<n>We show that unlikeliness reward mitigates rank bias and improves pass@$N$ across a large range of $N$ in both synthetic and real theorem proving settings.
  arXiv  Detail & Related papers  (2025-06-03T01:15:15Z)
• Redistributing Rewards Across Time and Agents for Multi-Agent Reinforcement Learning [14.852334980733369]
  Credit assignmen, disentangling each agent's contribution to a shared reward, is a critical challenge in cooperative multi-agent reinforcement learning.<n>We introduce Temporal-Agent Reward Redistribution (TAR$2$), an approach that decouples credit modeling from this constraint.<n>We demonstrate that this method is equivalent to a valid Potential-Based Reward Shaping (PBRS), which guarantees the optimal policy is preserved regardless of model accuracy.
  arXiv  Detail & Related papers  (2025-02-07T12:07:57Z)
• Truncating Trajectories in Monte Carlo Policy Evaluation: an Adaptive Approach [51.76826149868971]
  Policy evaluation via Monte Carlo simulation is at the core of many MC Reinforcement Learning (RL) algorithms. We propose as a quality index a surrogate of the mean squared error of a return estimator that uses trajectories of different lengths. We present an adaptive algorithm called Robust and Iterative Data collection strategy Optimization (RIDO)
  arXiv  Detail & Related papers  (2024-10-17T11:47:56Z)
• Interpretable Reward Redistribution in Reinforcement Learning: A Causal Approach [45.83200636718999]
  A major challenge in reinforcement learning is to determine which state-action pairs are responsible for future rewards that are delayed. We propose to explicitly model the contributions of state and action from a causal perspective, resulting in an interpretable reward redistribution. Experimental results show that our method outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-05-28T21:51:38Z)
• Low-variance estimation in the Plackett-Luce model via quasi-Monte Carlo sampling [58.14878401145309]
  We develop a novel approach to producing more sample-efficient estimators of expectations in the PL model. We illustrate our findings both theoretically and empirically using real-world recommendation data from Amazon Music and the Yahoo learning-to-rank challenge.
  arXiv  Detail & Related papers  (2022-05-12T11:15:47Z)
• Distributional Reinforcement Learning for Multi-Dimensional Reward Functions [91.88969237680669]
  We introduce Multi-Dimensional Distributional DQN (MD3QN) to model the joint return distribution from multiple reward sources. As a by-product of joint distribution modeling, MD3QN can capture the randomness in returns for each source of reward. In experiments, our method accurately models the joint return distribution in environments with richly correlated reward functions.
  arXiv  Detail & Related papers  (2021-10-26T11:24:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.