Process Reward Agents for Steering Knowledge-Intensive Reasoning

• URL: http://arxiv.org/abs/2604.09482v1
• Date: Fri, 10 Apr 2026 16:45:44 GMT
• Title: Process Reward Agents for Steering Knowledge-Intensive Reasoning
• Abstract summary: We introduce Process Reward Agents (PRA), a test-time method for providing domain-grounded, online, step-wise rewards to a frozen policy.<n>PRA consistently outperforms strong baselines, achieving 80.8% accuracy on MedQA with Qwen3-4B.<n> PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates.
• Score: 18.184546182516225
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Reasoning in knowledge-intensive domains remains challenging as intermediate steps are often not locally verifiable: unlike math or code, evaluating step correctness may require synthesizing clues across large external knowledge sources. As a result, subtle errors can propagate through reasoning traces, potentially never to be detected. Prior work has proposed process reward models (PRMs), including retrieval-augmented variants, but these methods operate post hoc, scoring completed trajectories, which prevents their integration into dynamic inference procedures. Here, we introduce Process Reward Agents (PRA), a test-time method for providing domain-grounded, online, step-wise rewards to a frozen policy. In contrast to prior retrieval-augmented PRMs, PRA enables search-based decoding to rank and prune candidate trajectories at every generation step. Experiments on multiple medical reasoning benchmarks demonstrate that PRA consistently outperforms strong baselines, achieving 80.8% accuracy on MedQA with Qwen3-4B, a new state of the art at the 4B scale. Importantly, PRA generalizes to unseen frozen policy models ranging from 0.5B to 8B parameters, improving their accuracy by up to 25.7% without any policy model updates. More broadly, PRA suggests a paradigm in which frozen reasoners are decoupled from domain-specific reward modules, allowing the deployment of new backbones in complex domains without retraining.

Related papers

• PRO-Step: Step-level Process Reward Optimization for Retrieval-Augmented Generation [2.198430261120653]
  We introduce PRO-STEP: we train a generative PRM that evaluates logical validity and evidential grounding at each step.<n>Experiments on single and multi-hop QA datasets demonstrate that PRO-STEP achieves the best average EM and F1 across five benchmarks.
  arXiv  Detail & Related papers  (2026-08-31T14:34:10Z)
• Mitigating Over-Optimization in PRM-Guided Search in Mathematical Reasoning by Optimizing the Guide [24.905102026459428]
  We show that PRM-guided search can over-optimize imperfect process rewards, pruning viable trajectories while expanding spurious ones.<n>We formulate the PRM-guided search as a robust optimization problem over plausible reward perturbations.<n>Without fine-tuning or online adaptation, maximin search consistently improves the PRM-guided search by 17-35% on average.
  arXiv  Detail & Related papers  (2026-08-30T21:25:03Z)
• The Hidden Bias of Process Reward Models:PRISM for Rewarding the Right Reasoning [54.0056619145783]
  Process Reward Models (PRMs) improve credit assignment for reasoning by providing step-level feedback.<n>We identify a hidden bias in PRMs caused by severe imbalance in step-level training data.<n>We propose PRISM, a policy-aware PRM training framework that learns from contrastive step-level comparisons.
  arXiv  Detail & Related papers  (2026-06-08T06:22:33Z)
• Back on Track: Aligning Rewards and States for Reasoning in Diffusion Large Language Models [90.09182925511317]
  Process Aligned Policy Optimization (PAPO) is a novel framework that holistically aligns the RL update with the dLLM's generative trajectory.<n>PAPO transforms sparse terminal rewards into dense, step-wise credit, and Entropy-Guided Historical Re-enactment (EHR) that replays authentic trajectories at high-uncertainty steps.
  arXiv  Detail & Related papers  (2026-06-07T07:59:55Z)
• Metacognition as Reward: Reinforcing LLM Reasoning via Knowledge and Regulation Signals [75.25256166997414]
  We introduce Metacognition-as-Reward (MaR), a metacognition-inspired RL framework.<n>MaR guides reasoning through two general process dimensions.<n>MaR consistently improves model performance, achieving up to a 7.7% gain over the base model.
  arXiv  Detail & Related papers  (2026-05-22T08:54:37Z)
• ODAR: Principled Adaptive Routing for LLM Reasoning via Active Inference [60.958331943869126]
  ODAR-Expert is an adaptive routing framework that optimize the accuracy-efficiency trade-off via principled resource allocation.<n>We show strong and consistent gains, including 98.2% accuracy on MATH and 54.8% on Humanity's Last Exam.
  arXiv  Detail & Related papers  (2026-02-27T05:22:01Z)
• ProRAG: Process-Supervised Reinforcement Learning for Retrieval-Augmented Generation [54.071574153853994]
  ProRAG is a process-supervised reinforcement learning framework designed to integrate learned step-level supervision into the online optimization loop.<n>Our framework consists of four stages: (1) Supervised Policy Warmup to initialize the model with a structured reasoning format; (2) construction of an MCTS-based Process Reward Model (PRM) to quantify intermediate reasoning quality; (3) PRM-Guided Reasoning Refinement to align the policy with fine-grained process preferences; and (4) Process-Supervised Reinforcement Learning with a dual-granularity advantage mechanism.
  arXiv  Detail & Related papers  (2026-01-29T16:04:59Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• BAPO: Boundary-Aware Policy Optimization for Reliable Agentic Search [72.87861928940929]
  Boundary-Aware Policy Optimization (BAPO) is a novel RL framework designed to cultivate reliable boundary awareness without compromising accuracy.<n>BAPO introduces two key components: (i) a group-based boundary-aware reward that encourages an IDK response only when the reasoning reaches its limit, and (ii) an adaptive reward modulator that strategically suspends this reward during early exploration, preventing the model from exploiting IDK as a shortcut.
  arXiv  Detail & Related papers  (2026-01-16T07:06:58Z)
• Agentic Reinforcement Learning with Implicit Step Rewards [92.26560379363492]
  Large language models (LLMs) are increasingly developed as autonomous agents using reinforcement learning (agentic RL)<n>We introduce implicit step rewards for agentic RL (iStar), a general credit-assignment strategy that integrates seamlessly with standard RL algorithms.<n>We evaluate our method on three challenging agent benchmarks, including WebShop and VisualSokoban, as well as open-ended social interactions with unverifiable rewards in SOTOPIA.
  arXiv  Detail & Related papers  (2025-09-23T16:15:42Z)
• Beyond Correctness: Harmonizing Process and Outcome Rewards through RL Training [26.589591658693962]
  Outcome Reward Models (ORMs) in RLVR are too coarse-grained to distinguish flawed reasoning within correct answers.<n>Process Reward Models (PRMs) offer fine-grained guidance for intermediate steps.<n>We introduce PRocess cOnsistency Filter (PROF) to harmonize noisy, fine-grained process rewards with accurate, coarse-grained outcome rewards.
  arXiv  Detail & Related papers  (2025-09-03T15:28:51Z)
• S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models [2.9925837108958864]
  Test-Time Scaling emerges as an active research focus in the large language model community.<n>Recent studies reveal that reasoning models (even Qwen3) consistently exhibit excessive thought redundancy.<n>This paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm.
  arXiv  Detail & Related papers  (2025-05-12T15:50:44Z)
• ReARTeR: Retrieval-Augmented Reasoning with Trustworthy Process Rewarding [25.329712997545794]
  We propose Retrieval-Augmented Reasoning through Trustworthy Process Rewarding (ReARTeR)<n>ReARTeR enhances RAG systems' reasoning capabilities through post-training and test-time scaling.<n> Experimental results on multi-step reasoning benchmarks demonstrate significant improvements.
  arXiv  Detail & Related papers  (2025-01-14T05:56:26Z)
• Rewarding Progress: Scaling Automated Process Verifiers for LLM Reasoning [90.23629291067763]
  A promising approach for improving reasoning in large language models is to use process reward models (PRMs) PRMs provide feedback at each step of a multi-step reasoning trace, potentially improving credit assignment over outcome reward models (ORMs) To improve a base policy by running search against a PRM or using it as dense rewards for reinforcement learning (RL), we ask: "How should we design process rewards?" We theoretically characterize the set of good provers and our results show that optimizing process rewards from such provers improves exploration during test-time search and online RL.
  arXiv  Detail & Related papers  (2024-10-10T17:31:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.