EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing

• URL: http://arxiv.org/abs/2606.00437v1
• Date: Sat, 30 May 2026 00:05:14 GMT
• Title: EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing
• Abstract summary: Process reward models (PRMs) are widely used in language-model training with dense step-level supervision.<n>We argue this assumption is not well validated.<n>We introduce textbfEST-PRM, a stress-testing framework for dense process rewards.
• Score: 6.324418485299426
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Process reward models (PRMs) are widely used in language-model training with dense step-level supervision. They assume PRM scores are stable proxies for step correctness under label-preserving transformations. These transformations change reasoning structure but preserve final answers. We argue this assumption is not well validated. Such transformations can change how PRM scores relate to correctness signals, leading to different failure modes across models.To address this gap, we introduce \textbf{EST-PRM}, a stress-testing framework for dense process rewards. It applies three transformations: (1) step inflation, (2) dependency-aware step reordering, and (3) confidence markers. A vulnerability decomposition is defined that separates reward inflation from loss of correctness sensitivity. Five PRM-style models are evaluated on 4,687 reasoning chains from MATH-500, GSM8K, and PRMBench.The results indicate clear differences in vulnerability patterns across models. Math-Shepherd shows the strongest sensitivity to position perturbations, with a Pearson correlation drop of $0.152 \pm 0.038$ and a $32.8 \pm 4.9\%$ score inflation rate. Qwen2.5-Math-PRM is most affected by step inflation, reaching a $47.6 \pm 4.3\%$ inflation rate. Confidence-based perturbations also distort reward calibration, revealing inconsistencies in correctness estimation. Three mitigation strategies are evaluated, highlighting trade-offs between robustness coverage and false-positive rates.

Related papers

• The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning [54.0056619145783]
  Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback.<n>We identify a hidden bias in PRMs caused by severe imbalance in step-level training data.<n>We propose PRISM, a policy-aware PRM training framework that learns from contrastive step-level comparisons.
  arXiv  Detail & Related papers  (2026-06-08T06:22:33Z)
• CEPO: RLVR Self-Distillation using Contrastive Evidence Policy Optimization [50.59956036193097]
  A model produces a correct solution under reinforcement learning with verifiable rewards (RLVR)<n>Every token receives the same reward signal regardless of whether it was a decisive reasoning step or a grammatical filler.<n>We propose Contrastive Evidence Policy Optimization (CEPO)<n>CEPO asks a sharper question at every token: "does the correct answer favor this token?" but "does the correct answer favor it while the wrong answer disfavors it?"
  arXiv  Detail & Related papers  (2026-05-19T06:46:19Z)
• Persona-Model Collapse in Emergent Misalignment [0.0]
  Fine-tuning large language models on narrow data with harmful content produces broadly misaligned behavior on unrelated prompts.<n>We test this hypothesis using two metrics: moral susceptibility (S) and moral robustness (R)<n>These metrics formalize the model's ability to differentiate characters (S) and its consistency when simulating a given one (R)
  arXiv  Detail & Related papers  (2026-05-13T00:48:57Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• Calibration Collapse Under Sycophancy Fine-Tuning: How Reward Hacking Breaks Uncertainty Quantification in LLMs [0.0]
  We investigate whether sycophantic reward signals degrade calibration.<n>Post-hoc matrix scaling applied to all three models reduces ECE by $40$--$64%$ and improves accuracy by $1.5$--$3.0$ percentage points.<n>These findings establish a methodology for evaluating the calibration impact of reward hacking and motivate calibration-aware training objectives.
  arXiv  Detail & Related papers  (2026-04-12T11:09:49Z)
• Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models [68.45272703833209]
  We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
  arXiv  Detail & Related papers  (2026-02-20T23:38:03Z)
• Adversarial Training for Process Reward Models [47.92183495904245]
  We introduce Adversarially Trained PRMs (textttAPRM), where a Generator ($G$) learns to produce reasoning errors to deceive a PRM ($R$)<n>This interaction yields progressively harder negatives for $R$, improving its robustness and generalization to novel errors without requiring manual step-level labels.
  arXiv  Detail & Related papers  (2025-11-28T05:32:01Z)
• Uncertainty in Language Models: Assessment through Rank-Calibration [65.10149293133846]
  Language Models (LMs) have shown promising performance in natural language generation. It is crucial to correctly quantify their uncertainty in responding to given inputs. We develop a novel and practical framework, termed $Rank$-$Calibration$, to assess uncertainty and confidence measures for LMs.
  arXiv  Detail & Related papers  (2024-04-04T02:31:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.