Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models

• URL: http://arxiv.org/abs/2603.06621v1
• Date: Fri, 20 Feb 2026 23:38:03 GMT
• Title: Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models
• Abstract summary: We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
• Score: 68.45272703833209
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Process Reward Models (PRMs) are rapidly becoming the backbone of LLM reasoning pipelines, yet we demonstrate that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure. To address this, we introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities. Static perturbation analysis uncovers a fluency-logic dissociation: high invariance to surface-level style changes reward changes $<$0.1, yet inconsistent detection of logically-corrupted reasoning, with different models failing on different attack types. Adversarial optimization demonstrates that gradient-based attacks inflate rewards on invalid trajectories, with reward landscapes exhibiting wide, exploitable peaks. RL-induced reward hacking exposes the critical failure mode: policies trained on AIME problems achieve near-perfect PRM rewards ($>$0.9), while ground-truth accuracy remains low (below 4%), with 43% of reward gains attributable to stylistic shortcuts. These findings reveal that current PRMs function as fluency detectors rather than reasoning verifiers, creating systematic blind spots that undermine their use as training signals. We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment. The code and dataset are available at https://github.com/SqueezeAILab/reward-under-attack.

Related papers

• From Viral to Void: Multi-Dimensional Behavioral and Contractual Analysis for Rug Pull Identification [2.7875224007687827]
  Rug Pull scams are characterized by extremely low deployment costs, covert execution, rapid fund transfers, and high detection difficulty.<n>This paper develops a multi-dimensional feature system based on dimensions such as malicious smart contract design, on-chain transaction anomalies, liquidity manipulation, and social media disclosures.<n> Experiments show that the model achieves an accuracy of 0.927, an F1 score of 0.787, and an AUC-ROC of 0.952 on the test set, outperforming traditional methods.
  arXiv  Detail & Related papers  (2026-08-03T02:34:24Z)
• EST-PRM: Stress-Testing Process Reward Models Before They Become Load-Bearing [6.324418485299426]
  Process reward models (PRMs) are widely used in language-model training with dense step-level supervision.<n>We argue this assumption is not well validated.<n>We introduce textbfEST-PRM, a stress-testing framework for dense process rewards.
  arXiv  Detail & Related papers  (2026-05-30T00:05:14Z)
• Unleashing Implicit Rewards: Prefix-Value Learning for Distribution-Level Optimization [74.91418266859297]
  Implicit process reward models (PRMs) provide fine-grained reward signals along the reasoning process.<n>PRMs suffer from a train-inference mismatch: training only constrains a sequence-level aggregate, whereas inference requires token-level scores to reflect local step quality.<n>We propose a novel Implicit Prefix-Value Reward Model (IPVRM), which directly learns a prefix-conditioned value function estimating the probability of eventual correctness.<n>We also propose Distribution-Level RL (DistRL), which computes TD advantages for both sampled tokens and high-probability candidate tokens.
  arXiv  Detail & Related papers  (2026-04-14T18:19:54Z)
• IR$^3$: Contrastive Inverse Reinforcement Learning for Interpretable Detection and Mitigation of Reward Hacking [67.20568716300272]
  Reinforcement Learning from Human Feedback (RLHF) enables powerful LLM alignment but can introduce reward hacking.<n>We introduce IR3 (Interpretable Reward Reconstruction and Rectification), a framework that reverse-engineers, interprets, and surgically repairs the implicit objectives driving RLHF-tuned models.<n>We show that IR3 achieves 0.89 correlation with ground-truth rewards, identifies hacking features with over 90% precision, and significantly reduces hacking behaviors while maintaining capabilities within 3% of the original model.
  arXiv  Detail & Related papers  (2026-02-23T01:14:53Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• CS-GBA: A Critical Sample-based Gradient-guided Backdoor Attack for Offline Reinforcement Learning [7.5200963577855875]
  Offline Reinforcement Learning (RL) enables policy optimization from static datasets but is inherently vulnerable to backdoor attacks.<n>We propose CS-GBA (Critical Sample-based Gradient-guided Backdoor Attack), a novel framework designed to achieve high stealthiness and destructiveness under a strict budget.
  arXiv  Detail & Related papers  (2026-01-15T13:57:52Z)
• Information-Theoretic Reward Modeling for Stable RLHF: Detecting and Mitigating Reward Hacking [78.69179041551014]
  We propose an information-theoretic reward modeling framework based on the Information Bottleneck principle.<n>We show that InfoRM filters out preference-irrelevant information to alleviate reward misgeneralization.<n>We also introduce IBL, a distribution-level regularization that penalizes such deviations, effectively expanding the optimization landscape.
  arXiv  Detail & Related papers  (2025-10-15T15:51:59Z)
• One Token Embedding Is Enough to Deadlock Your Large Reasoning Model [91.48868589442837]
  We present the Deadlock Attack, a resource exhaustion method that hijacks an LRM's generative control flow.<n>Our method achieves a 100% attack success rate across four advanced LRMs.
  arXiv  Detail & Related papers  (2025-10-12T07:42:57Z)
• Causal Reward Adjustment: Mitigating Reward Hacking in External Reasoning via Backdoor Correction [5.518813485456855]
  External reasoning systems combine language models with process reward models (PRMs) to select high-quality reasoning paths for complex tasks.<n>These systems are prone to reward hacking, where high-scoring but logically incorrect paths are assigned high scores by the PRMs, leading to incorrect answers.<n>We propose Causal Reward Adjustment (CRA), a method that mitigates reward hacking by estimating the true reward of a reasoning path.
  arXiv  Detail & Related papers  (2025-08-06T08:48:55Z)
• One Token to Fool LLM-as-a-Judge [52.45386385722788]
  Large language models (LLMs) are increasingly trusted as automated judges, assisting evaluation and providing reward signals for training other models.<n>We uncover a critical vulnerability even in this reference-based paradigm: generative reward models are systematically susceptible to reward hacking.
  arXiv  Detail & Related papers  (2025-07-11T17:55:22Z)
• Stop Summation: Min-Form Credit Assignment Is All Process Reward Model Needs for Reasoning [36.320502608626896]
  Process reward models (PRMs) have proven effective for test-time scaling of Large Language Models (LLMs) on challenging reasoning tasks.<n>However, reward hacking issues with PRMs limit their successful application in reinforcement fine-tuning.<n>In this paper, we identify the main cause of PRM-induced reward hacking: the canonical summation-form credit assignment in reinforcement learning.
  arXiv  Detail & Related papers  (2025-04-21T17:59:02Z)
• Probabilistic Uncertain Reward Model [27.40414952747553]
  This paper proposes the Probabilistic Uncertain Reward Model (PURM) to learn the reward distributions that emerged from the preference data.<n>We show that PURM outperforms existing methods with more accurate reward and sound uncertainty estimations.
  arXiv  Detail & Related papers  (2025-03-28T14:39:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.