The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment

• URL: http://arxiv.org/abs/2606.27739v1
• Date: Fri, 26 Jun 2026 05:38:08 GMT
• Title: The Weakest Link Tells It All: Outcome-Supervised Process Reward Modeling via Learnable Credit Assignment
• Abstract summary: Process reward models (PRMs) enhance the reasoning capabilities of large language models (LLMs)<n>Outcome-supervised PRMs offer a scalable alternative by learning from final-answer correctness alone.<n>We propose Outcome-Supervised Process Reward Modeling via **L**earnable **C**redit **A**ssignment (**LCA**)
• Score: 25.463488253598133
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Process reward models (PRMs) enhance the reasoning capabilities of large language models (LLMs) by providing fine-grained feedback, yet training PRMs typically requires expensive stepwise annotations. Outcome-supervised PRMs offer a scalable alternative by learning from final-answer correctness alone, but this introduces a fundamental *credit assignment* challenge, i.e., attributing outcomes to responsible reasoning steps. Existing approaches rely on either uniform or causal assignment, both of which fail to anchor credit in step correctness and thus hinder process error identification. In this work, we propose Outcome-Supervised Process Reward Modeling via **L**earnable **C**redit **A**ssignment (**LCA**), an outcome-supervised PRM framework that jointly learns credit assignment and reward modeling under the principle of *Weakest Link Assignment: a reasoning chain is as strong as its weakest link*. To address mutual dependence between credit assignment and reward modeling, we formalize outcome-supervised PRM as a Multiple Instance Learning (MIL) problem and introduce Softmax-Weighted-Sum (SWS) pooling, an MIL pooling technique tailored for strong dependence and redundancy among reasoning states. We prove Bayes consistency of our algorithm under mild assumptions. Extensive experiments demonstrate that **LCA** consistently outperforms state-of-the-art outcome-supervised PRMs across multiple tasks and backbones. Code is available at https://anonymous.4open.science/r/LCA.

Related papers

• Unsupervised Process Reward Models [22.283171054858496]
  Process Reward Models (PRMs) are powerful mechanisms for steering large language model reasoning.<n>PRMs require expert annotations for every reasoning step, making them costly and difficult to scale.<n>Here, we propose a method for training unsupervised PRMs that requires no human supervision, neither at the level of step-by-step annotations nor through ground-truth verification of final answers.
  arXiv  Detail & Related papers  (2026-05-11T08:05:27Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• DPRM: A Dual Implicit Process Reward Model in Multi-Hop Question Answering [62.11407895095892]
  In question answering tasks, Chain of Thought (CoT) guides large language models through multi-step reasoning, and Knowledge Graphs (KGs) reduce hallucinations via semantic matching.<n>Traditional Process Reward Models (PRMs) evaluate the reasoning process but require costly human annotations or rollout generation.<n>We propose the DPRM (Dual Implicit Process Reward Model), which trains two implicit PRMs for CoT and KG reasoning in MHQA tasks.
  arXiv  Detail & Related papers  (2025-11-11T15:41:22Z)
• GroundedPRM: Tree-Guided and Fidelity-Aware Process Reward Modeling for Step-Level Reasoning [34.42899160708635]
  We introduce GroundedPRM, a tree-guided and fidelity-aware framework for automatic process supervision.<n>GroundedPRM is trained on only 40K automatically labeled samples, amounting to just 10% of the data used by the best-performing PRM trained with auto-labeled supervision.<n>It achieves up to a 26% relative improvement in average performance on ProcessBench.
  arXiv  Detail & Related papers  (2025-10-16T17:54:07Z)
• From <Answer> to <Think>: Multidimensional Supervision of Reasoning Process for LLM Optimization [62.07990937720985]
  Dimension-level Reward Model (DRM) is a new supervision framework for Large Language Models.<n>DRM evaluates the quality of a reasoning process along three fundamental, complementary, and interpretable dimensions.<n> Experimental results show that DRM provides effective supervision signals, guides the optimization of LLMs and enhances their reasoning ability.
  arXiv  Detail & Related papers  (2025-10-13T14:29:15Z)
• Linking Process to Outcome: Conditional Reward Modeling for LLM Reasoning [30.302863491794543]
  Process Reward Models (PRMs) aim to guide their step-by-step reasoning toward a final answer.<n>Existing PRMs fail to capture inter-step dependencies, or struggle to align process rewards with the final outcome.<n>We propose Conditional Reward Modeling that frames reasoning as a temporal process leading to a correct answer.
  arXiv  Detail & Related papers  (2025-09-30T17:38:45Z)
• GM-PRM: A Generative Multimodal Process Reward Model for Multimodal Mathematical Reasoning [12.724393910603299]
  We introduce the Generative Multimodal Process Reward Model (GM-PRM)<n>Instead of a simple scalar score, GM-PRM provides a fine-grained, interpretable analysis of each reasoning step.<n>We show that GM-PRM achieves state-of-the-art results on multiple multimodal math benchmarks.
  arXiv  Detail & Related papers  (2025-08-06T05:10:29Z)
• Discriminative Policy Optimization for Token-Level Reward Models [55.98642069903191]
  Process reward models (PRMs) provide more nuanced supervision compared to outcome reward models (ORMs)<n>Q-RM explicitly learns token-level Q-functions from preference data without relying on fine-grained annotations.<n>Reinforcement learning with Q-RM significantly enhances training efficiency, achieving convergence 12 times faster than ORM on GSM8K and 11 times faster than step-level PRM on MATH.
  arXiv  Detail & Related papers  (2025-05-29T11:40:34Z)
• Reward Models Identify Consistency, Not Causality [54.987590763737145]
  State-of-the-art reward models prioritize structural consistency over causal correctness.<n>Removing the problem statement has minimal impact on reward scores.<n> altering numerical values or disrupting the reasoning flow significantly affects RM outputs.
  arXiv  Detail & Related papers  (2025-02-20T14:57:14Z)
• Free Process Rewards without Process Labels [55.14044050782222]
  We show that an textitimplicit PRM can be obtained at no additional cost, by simply training an ORM on the cheaper response-level labels.<n>We show that our implicit PRM, when instantiated with the cross-entropy (CE) loss, is more data-efficient and can keep improving generation models even when trained with only one response per instruction.
  arXiv  Detail & Related papers  (2024-12-02T21:20:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.