Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certify

• URL: http://arxiv.org/abs/2608.08008v1
• Date: Sat, 08 Aug 2026 08:41:06 GMT
• Title: Quality-Diversity Stress Tests for Process Reward Models:What Archive Coverage Can and Cannot Certify
• Abstract summary: We formulate PRM stress testing as a quality-diversity search problem using MAP-Elites.<n>We characterize what such archives certify: finite-cell repair bounds covered-cell tail risk and average residual severity but cannot bound the worst remaining cell from covered fraction alone.<n>A predeclared paired LoRA repair protocol reduces exploit rates from 0.148 to 0.037 to 0.074, the worst attack from 0.333 to 0.177 to 0.212, improves ranking AUROC without degrading best-of-4 accuracy.
• Score: 1.0742675209112622
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Process reward models (PRMs) score intermediate reasoning steps and are widely used for search, ranking, and training, but optimization can exploit these learned proxies by increasing reward while turning correct reasoning into incorrect reasoning. We formulate PRM stress testing as a quality-diversity search problem using MAP-Elites, retaining the most severe correctness-flipping edit in each behavior-space region while separating search coverage from exploit coverage. We characterize what such archives certify: finite-cell repair bounds covered-cell tail risk and average residual severity but cannot bound the worst remaining cell from covered fraction alone; under Lipschitz post-repair loss and metric-cover auditing, the residual is bounded by archive fitting error plus the Lipschitz constant times the covering radius. A controlled landscape validates this certificate and the impossibility of any fraction-only worst-case guarantee. On real PRMs, the search reveals an aggregation-dependent vulnerability in Qwen2.5-Math-PRM-7B: padding yields 44 strict exploits with maximum gain 0.294 under mean pooling versus one exploit under minimum readout; a matched syntactic control isolates the mechanism, and an RLHFlow value-head model shows the same qualitative effect with maximum gain 0.005. A predeclared paired LoRA repair protocol reduces exploit rates from 0.148 to 0.037 to 0.074, lowers the worst attack from 0.333 to 0.177 to 0.212, improves ranking AUROC without degrading best-of-4 accuracy, attributes gains to adversarial fine-tuning rather than archive diversity, and is confirmed by independent unpaired replications (44 to 1, clean-split worst gain 0.0092, MATH-500 41 to 0, clean ranking 40/40).

Related papers

• Valid Per-Field Selective Risk Control for Document Extraction: Three Failure Modes, a Validity Ladder, and When Conditioning Pays [0.0]
  Per-field accept/review with selective risk at most alpha is the trust contract document-extraction systems need.<n>On 13,859 genuine claude-sonnet-5 fields from 800 CORD receipts (49.0% correct) we diagnose three failure modes.<n>Released Apache-2.0 with seed-pinned, regression-gated procedures.
  arXiv  Detail & Related papers  (2026-07-28T17:08:50Z)
• Mapping CVEs to MITRE ATT&CK Techniques: A Curated Gold-Set Classifier and the Limits of LLM-Assisted Label Expansion [46.262619407930266]
  We train a multi-label classifier on a curated gold dataset of 1,207 CVEs from expert MITRE Center for Threat-Informed Defense mappings.<n>The resulting model approximately doubles recall@5 compared with a zero-shot embedding-similarity baseline.<n>We then investigate whether LLM-assisted labeling can extend the gold dataset.
  arXiv  Detail & Related papers  (2026-07-28T10:59:04Z)
• Safety-Aware Cascaded Inference for Crop Damage Assessment with Controlled Error Trade-offs [0.0]
  In picture-based agricultural insurance for smallholder farmers, missed damage detections carry substantially higher cost than false alarms.<n>We propose CascadeCropNet, a two-stage cascade architecture calibrated to satisfy a target recall constraint.<n>A lightweight Sentinel model performs binary health triage; samples exceeding a calibrated damage probability threshold tau are escalated to a specialist Expert model for fine-grained diagnosis.
  arXiv  Detail & Related papers  (2026-07-28T09:01:27Z)
• Looping Is Not Reliability: State-Bound Evidence and Typed Revision Contracts for Agentic Code Repair [36.56438114281786]
  We study the gap between finding a correct patch and retaining, verifying, and submitting it.<n>We derive an evidence-bound typed loop contract and instantiate its mechanically enforceable subset in a reference implementation.
  arXiv  Detail & Related papers  (2026-07-27T16:05:23Z)
• RUBRIC: Realism--Utility Balanced Ranking for Imbalanced Classification [38.8278639079174]
  Class imbalance poses a fundamental challenge in risk-sensitive applications such as fraud detection and medical diagnosis.<n>We introduce RUBRIC, a generator-agnostic filtering framework that formulates synthetic sample selection as a quality-over-quantity optimization problem.<n>We show that RUBRIC improves F1-macro and recall while maintaining comparable ROC-AUC across several generators.
  arXiv  Detail & Related papers  (2026-07-10T06:37:12Z)
• Distributional Energy-Based Models for Uncertainty-Aware Structured LLM Reasoning [40.342912574072024]
  Large Language Models produce structured outputs such as travel plans or code solutions.<n>Individual reasoning steps may appear correct while the output as a whole violates budgets, fails test cases, or contradicts earlier deductions.<n>We propose a learned quality scorer with deterministic analytical constraint penalties for verifying structured LLM outputs.
  arXiv  Detail & Related papers  (2026-05-15T17:08:27Z)
• RVPO: Risk-Sensitive Alignment via Variance Regularization [13.192921543523283]
  We propose Reward-Variance Policy Optimization (RVPO), a risk-sensitive framework that penalizes inter-reward variance during advantage aggregation.<n>We show via Taylor expansion that a LogSumExp (SoftMin) operator effectively acts as a smooth variance penalty.<n>By preventing the model from neglecting difficult constraints to exploit easier objectives, RVPO improves overall scores on HealthBench.
  arXiv  Detail & Related papers  (2026-05-07T06:43:05Z)
• Predictive Entropy Links Calibration and Paraphrase Sensitivity in Medical Vision-Language Models [2.064612766965483]
  We show five uncertainty quantification methods on MedGemma 4BIT across in distribution MIMIC CXR and outof distribution PadChest chest X ray datasets, with cross architecture validation on LLaVA RAD7B.<n>For well calibrated single model methods, predictive entropy from one forward pass predicts which samples will flip under rephrasing AUROC 0.711 on MedGemma, 0.878 on LLaVARAD p 10 4, enabling a single entropy threshold to flag both unreliable and sensitive predictions.
  arXiv  Detail & Related papers  (2026-04-10T04:18:47Z)
• Reward Under Attack: Analyzing the Robustness and Hackability of Process Reward Models [68.45272703833209]
  We show that state-of-the-art PRMs are systematically exploitable under adversarial optimization pressure.<n>We introduce a three-tiered diagnostic framework that applies increasing adversarial pressure to quantify these vulnerabilities.<n>We release PRM-BiasBench and a diagnostic toolkit to enable robustness evaluation before deployment.
  arXiv  Detail & Related papers  (2026-02-20T23:38:03Z)
• CARE What Fails: Contrastive Anchored-REflection for Verifiable Multimodal [84.71254539482369]
  Group-relative reinforcement learning with verifiable rewards (RLVR) often wastes the most informative data it already has the failures.<n>We present CARE, a failure-centric post-training framework for multimodal reasoning that turns errors into supervision.<n> CARE improves accuracy and training smoothness while explicitly increasing the share of learning signal that comes from failures.
  arXiv  Detail & Related papers  (2025-12-22T16:34:21Z)
• Causally-Enhanced Reinforcement Policy Optimization [36.523007244998695]
  Causally-Enhanced Policy Optimization (CE-PO) is a drop-in reward-shaping framework that augments policy optimization with a differentiable proxy for causal coherence.<n>CE-PO estimates model-internal influence with Jacobian-based sensitivities, counterfactually hardens these signals to suppress nuisance cues, and fuses the resulting coherence score with task-accuracy feedback.<n> Experimental results across 4 datasets show that CE-PO improves accuracy over baselines by 5.49% on average (up to 9.58%), while improving robustness to correlation-causation flips and light counterfactual edits.
  arXiv  Detail & Related papers  (2025-09-27T04:10:16Z)
• Probabilistic Soundness Guarantees in LLM Reasoning Chains [37.440902632372904]
  Autoregressive Reasoning Entailment Stability (ARES) is a probabilistic framework that evaluates each reasoning step based solely on previously-verified premises.<n>ARES achieves state-of-the-art performance across four benchmarks and demonstrates superior robustness on very long synthetic reasoning chains.
  arXiv  Detail & Related papers  (2025-07-17T09:40:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.