Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training

• URL: http://arxiv.org/abs/2604.01597v1
• Date: Thu, 02 Apr 2026 04:09:31 GMT
• Title: Learning from the Right Rollouts: Data Attribution for PPO-based LLM Post-Training
• Abstract summary: We propose textbfInfluence-Guided PPO (I-PPO), a novel framework that integrates data attribution into the RL post-training loop.<n>I-PPO consistently outperforms SFT and PPO baselines.
• Score: 25.736034135647554
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Traditional RL algorithms like Proximal Policy Optimization (PPO) typically train on the entire rollout buffer, operating under the assumption that all generated episodes provide a beneficial optimization signal. However, these episodes frequently contain noisy or unfaithful reasoning, which can degrade model performance and slow down training. In this paper, we propose \textbf{Influence-Guided PPO (I-PPO)}, a novel framework that integrates data attribution into the RL post-training loop. By calculating an influence score for each episode using a gradient-based approximation, I-PPO identifies and eliminates episodes that are anti-aligned with a validation gradient. Our experiments demonstrate that I-PPO consistently outperforms SFT and PPO baselines. We show that our filtering process acts as an intrinsic early stopping mechanism, accelerating training efficiency while effectively reducing unfaithful CoT reasoning.

Related papers

• PS-PPO: Prefix-Sampling PPO for Critic-Free RLHF [20.068789951726078]
  Reinforcement Learning from Human Feedback is a practical alternative to actor--critic training.<n>Existing critic-free approaches propagate a trajectory-level learning signal uniformly across all tokens in a trajectory.<n>We propose Prefix-Sampling Proximal Policy Optimization (PS-PPO), a compute-efficient critic-free method for RLHF.
  arXiv  Detail & Related papers  (2026-06-29T04:04:32Z)
• Unbiased Dynamic Pruning for Efficient Group-Based Policy Optimization [60.87651283510059]
  Group Relative Policy Optimization (GRPO) effectively scales LLM reasoning but incurs prohibitive computational costs.<n>We propose Dynamic Pruning Policy Optimization (DPPO), a framework that enables dynamic pruning while preserving unbiased gradient estimation.<n>To mitigate the data sparsity induced by pruning, we introduce Dense Prompt Packing, a window-based greedy strategy.
  arXiv  Detail & Related papers  (2026-03-04T14:48:53Z)
• Rethinking the Trust Region in LLM Reinforcement Learning [72.25890308541334]
  Proximal Policy Optimization (PPO) serves as the de facto standard algorithm for Large Language Models (LLMs)<n>We propose Divergence Proximal Policy Optimization (DPPO), which substitutes clipping with a more principled constraint.<n>DPPO achieves superior training and efficiency compared to existing methods, offering a more robust foundation for RL-based fine-tuning.
  arXiv  Detail & Related papers  (2026-02-04T18:59:04Z)
• Coverage Improvement and Fast Convergence of On-policy Preference Learning [67.36750525893514]
  Online on-policy preference learning algorithms for language model alignment can significantly outperform their offline counterparts.<n>We analyze how the sampling policy's coverage evolves throughout on-policy training.<n>We develop principled on-policy schemes for reward distillation in the general function class setting.
  arXiv  Detail & Related papers  (2026-01-13T10:46:06Z)
• Truncated Proximal Policy Optimization [43.965892659920364]
  Truncated Proximal Policy Optimization (T-PPO) improves training efficiency by streamlining policy update and length-restricted response generation.<n>We propose Extended Generalized Advantage Estimation (EGAE) for advantage estimation derived from incomplete responses.<n>We demonstrate the effectiveness and efficacy of T-PPO on AIME 2024 with a 32B base model.
  arXiv  Detail & Related papers  (2025-06-18T01:21:38Z)
• CPPO: Accelerating the Training of Group Relative Policy Optimization-Based Reasoning Models [77.16976971950785]
  This paper introduces Completion Pruning Policy Optimization (CPPO) to accelerate the training of reasoning models.<n>CPPO prunes completions with low absolute advantages, significantly reducing the number needed for gradient calculation and updates.<n> Experiments show that CPPO achieves up to $7.98times$ speedup on GSM8K and $3.48times$ on Math while preserving or even enhancing the accuracy compared to the original GRPO.
  arXiv  Detail & Related papers  (2025-03-28T11:30:05Z)
• What's Behind PPO's Collapse in Long-CoT? Value Optimization Holds the Secret [3.410112345043215]
  We propose Value-Calibrated PPO (VC-PPO) to address these issues.<n>Experiments on the American Invitational Mathematics Examination (AIME) show that VC-PPO significantly boosts PPO performance.
  arXiv  Detail & Related papers  (2025-03-03T12:59:25Z)
• Lean and Mean: Decoupled Value Policy Optimization with Global Value Guidance [52.65461207786633]
  Policy-based Reinforcement Learning from Human Feedback is essential for aligning large language models with human preferences.<n>It requires joint training of an actor and critic with a pretrained, fixed reward model for guidance.<n>We propose textbfDecoupled Value Policy Optimization (DVPO), a lean framework that replaces traditional reward modeling with a pretrained emphglobal value model (GVM)
  arXiv  Detail & Related papers  (2025-02-24T08:11:33Z)
• Is DPO Superior to PPO for LLM Alignment? A Comprehensive Study [16.99550556866219]
  Reinforcement Learning from Human Feedback (RLHF) is currently the most widely used method to align large language models (LLMs) with human preferences. In academic benchmarks, state-of-the-art results are often achieved via reward-free methods, such as Direct Preference Optimization (DPO) We show that PPO is able to surpass other alignment methods in all cases and achieve state-of-the-art results in challenging code competitions.
  arXiv  Detail & Related papers  (2024-04-16T16:51:53Z)
• A dynamical clipping approach with task feedback for Proximal Policy Optimization [29.855219523565786]
  There is no theoretical proof that the optimal PPO clipping bound remains consistent throughout the entire training process. Past studies have aimed to dynamically adjust PPO clipping bound to enhance PPO's performance. We propose Preference based Proximal Policy Optimization (Pb-PPO) to better reflect the preference (maximizing Return) of reinforcement learning tasks.
  arXiv  Detail & Related papers  (2023-12-12T06:35:56Z)
• You May Not Need Ratio Clipping in PPO [117.03368180633463]
  Proximal Policy Optimization (PPO) methods learn a policy by iteratively performing multiple mini-batch optimization epochs of a surrogate objective with one set of sampled data. Ratio clipping PPO is a popular variant that clips the probability ratios between the target policy and the policy used to collect samples. We show in this paper that such ratio clipping may not be a good option as it can fail to effectively bound the ratios. We show that ESPO can be easily scaled up to distributed training with many workers, delivering strong performance as well.
  arXiv  Detail & Related papers  (2022-01-31T20:26:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.