The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning

• URL: http://arxiv.org/abs/2606.09078v1
• Date: Mon, 08 Jun 2026 06:22:33 GMT
• Title: The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning
• Abstract summary: Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback.<n>We identify a hidden bias in PRMs caused by severe imbalance in step-level training data.<n>We propose PRISM, a policy-aware PRM training framework that learns from contrastive step-level comparisons.
• Score: 54.0056619145783
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback. However, we identify a hidden bias in PRMs caused by severe imbalance in step-level training data. Standard cross-entropy training amplifies this bias, causing PRMs to overcredit plausible but incorrect steps and produce high false-positive rates. We show that these false positives have an asymmetric downstream effect: false negatives mainly slow exploration, whereas false positives actively steer Best-of-N selection, guided decoding, and policy optimization toward flawed reasoning. This suggests that PRM training should shift from pointwise label fitting to reliable relative comparisons. To address this, we propose PRISM (Precision Ranking for Improved Step Modeling), a policy-aware PRM training framework that learns from contrastive step-level comparisons and hard negatives generated by a temporal lookahead strategy, requiring no new human labels. We further use a difficulty-aware curriculum to optimize the contrastive step margin. Across PRMBench and ProcessBench, PRISM substantially reduces false positives (22% on PRMBench) and improves macro F1 over strong discriminative PRMs. When applied to policy optimization and search tasks, including guided decoding and Best-of-N selection, it consistently improves accuracy (up to 22% for guided decoding and 33% for Best-of-N) and robustness. More broadly, trustworthy process supervision is not just about assigning high rewards, but about rewarding the right reasoning for the right reasons.

Related papers

• Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide [24.905102026459428]
  We show that PRM-guided search can over-optimize imperfect process rewards, pruning viable trajectories while expanding spurious ones.<n>We formulate the PRM-guided search as a robust optimization problem over plausible reward perturbations.<n>Without fine-tuning or online adaptation, maximin search consistently improves the PRM-guided search by 17-35% on average.
  arXiv  Detail & Related papers  (2026-08-30T21:25:03Z)
• Unsupervised Process Reward Models [22.283171054858496]
  Process Reward Models (PRMs) are powerful mechanisms for steering large language model reasoning.<n>PRMs require expert annotations for every reasoning step, making them costly and difficult to scale.<n>Here, we propose a method for training unsupervised PRMs that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers.
  arXiv  Detail & Related papers  (2026-05-11T08:05:27Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• LLM Reasoning with Process Rewards for Outcome-Guided Steps [7.3312414850609215]
  We introduce process reward models (PRMs) to score intermediate steps and provide denser supervision.<n>PRM scores are often imperfectly aligned with final correctness and can reward locally fluent reasoning that still ends in an incorrect answer.<n>We propose PROGRS, a framework that leverages PRMs while keeping outcome correctness dominant.
  arXiv  Detail & Related papers  (2026-02-08T06:38:20Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• Towards Robust Process Reward Modeling via Noise-aware Learning [33.1289107681179]
  We propose a two-stage framework to mitigate noisy supervision.<n>In the labeling stage, we introduce a reflection-aware label correction mechanism that uses a large language model (LLM) as a judge.<n>In the training stage, we propose a underlinetextbfIterative underlinetextbfTraining framework that enables the PRM to progressively refine noisy labels.
  arXiv  Detail & Related papers  (2026-01-19T06:03:58Z)
• Beyond the First Error: Process Reward Models for Reflective Mathematical Reasoning [49.21525229904197]
  We propose a novel data annotation method for PRMs specifically designed to score the long CoT reasoning process.<n>We introduce the concepts of Error Propagation and Error Cessation, enhancing PRMs' ability to identify both effective self-correction behaviors and reasoning based on erroneous steps.<n>Our PRM achieves superior performance across various metrics, including search guidance, BoN, and F1 scores.
  arXiv  Detail & Related papers  (2025-05-20T14:12:05Z)
• Free Process Rewards without Process Labels [55.14044050782222]
  We show that an textitimplicit PRM can be obtained at no additional cost, by simply training an ORM on the cheaper response-level labels.<n>We show that our implicit PRM, when instantiated with the cross-entropy (CE) loss, is more data-efficient and can keep improving generation models even when trained with only one response per instruction.
  arXiv  Detail & Related papers  (2024-12-02T21:20:02Z)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning [90.23629291067763]
  A promising approach for improving reasoning in large language models is to use process reward models (PRMs) PRMs provide feedback at each step of a multi-step reasoning trace, potentially improving credit assignment over outcome reward models (ORMs) To improve a base policy by running search against a PRM or using it as dense rewards for reinforcement learning (RL), we ask: "How should we design process rewards?" We theoretically characterize the set of good provers and our results show that optimizing process rewards from such provers improves exploration during test-time search and online RL.
  arXiv  Detail & Related papers  (2024-10-10T17:31:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.