ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying

• URL: http://arxiv.org/abs/2609.04648v1
• Date: Fri, 04 Sep 2026 02:29:10 GMT
• Title: ConsensusBench: Benchmark of Consensus Nodes for LLM Reasoning via Outcome Reward Densifying
• Abstract summary: We develop a new reinforcement learning signal named ConsensusPR.<n>It directly reduces the reward sparsity of outcome reward across long reasoning trajectories.<n> Experiments across AIME 2024, AIME 2025, GSM8K, MATH-500, and our ConsensusBench demonstrate that the proposed method consistently surpasses GRPO-style approaches.
• Score: 45.2850302643347
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reinforcement learning (RL) has become one of the primary paradigms for reasoning enhancement of large language models (LLMs). In particular, Group Relative Policy Optimization (GRPO) and related algorithms have demonstrated strong performance with outcome-level rewards. However, these methods depend solely on the final answer, without feedback regarding which intermediate steps contribute to success or failure. As task complexity and reasoning trajectory length increase, such sparse final-answer rewards become increasingly insufficient. To address this limitation, we introduce ConsensusBench, a novel dataset designed to provide rule-based process-level signals. We posit that a correct final answer relies on a small set of intermediate conclusions throughout the reasoning process, which can be seen as a verifiable sub-outcome. We identify these sub-outcomes by filtering correct trajectories from N rollouts and clustering semantically equivalent intermediate statements. We call these clustered statements as Consensus Nodes. By integrating a rule-based process reward derived from these nodes into GRPO-style algorithms, we develop a new reinforcement learning signal named ConsensusPR. It directly reduces the reward sparsity of outcome reward across long reasoning trajectories. To facilitate systematic process-level evaluation, we introduce three metrics to our benchmark: Final Answer Accuracy (Acc), Node Coverage Rate (NCR), and Tokens per Node (TPN). Experiments across AIME 2024, AIME 2025, GSM8K, MATH-500, and our ConsensusBench demonstrate that the proposed method consistently surpasses GRPO-style approaches, highlighting the practical value of consensus nodes in guiding reasoning.

Related papers

• N-GRPO: Embedding-Level Neighbor Mixing for Enhanced Policy Optimization [55.14402862283128]
  We introduce N-GRPO, a novel exploration strategy integrated into the Group Relative Policy Optimization framework.<n>Rather than relying on token-level sampling or native embedding-level noise, our approach leverages Semantic Neighbor Mixing.<n>N-GRPO achieves consistent improvements over strong baselines on math reasoning benchmarks and also exhibits robust generalization capabilities on out-of-distribution tasks.
  arXiv  Detail & Related papers  (2026-06-09T12:21:27Z)
• From Failure to Feedback: Group Revision Unlocks Hard Cases in Object-Level Grounding [35.008790179255136]
  We propose a group-revision optimisation paradigm that enhances learning on hard cases.<n>Inspired by reward shaping, we introduce a consolidation process that quantifies each candidate's improvement over the initial attempt.<n>Our method achieves consistent gains across referring and reasoning segmentation, REC, and counting benchmarks compared with prior GRPO-based models.
  arXiv  Detail & Related papers  (2026-05-15T13:41:41Z)
• Conditional Advantage Estimation for Reinforcement Learning in Large Reasoning Models [50.84995206660551]
  We introduce Conditional advANtage estimatiON (CANON) to amplify the impact of a target metric without presuming its direction.<n>CANON based on entropy consistently outperforms prior methods on both math reasoning and high-complexity logic tasks.
  arXiv  Detail & Related papers  (2025-09-28T16:33:07Z)
• Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling [12.835376812101323]
  We introduce the hypothesis that PRMs are also Partial Reward Models.<n>This allows for principled early rejection based on intermediate token-level signals.<n>On math reasoning benchmarks, our method achieves up to 1.4$times$-9$times$ reduction in inference FLOPs without degrading final performance.
  arXiv  Detail & Related papers  (2025-08-04T00:58:56Z)
• Intra-Trajectory Consistency for Reward Modeling [67.84522106537274]
  We develop an intra-trajectory consistency regularization to enforce that adjacent processes with higher next-token generation probability maintain more consistent rewards.<n>We show that the reward model trained with the proposed regularization induces better DPO-aligned policies and achieves better best-of-N (BON) inference-time verification results.
  arXiv  Detail & Related papers  (2025-06-10T12:59:14Z)
• TreeRPO: Tree Relative Policy Optimization [65.51935468270916]
  name is a novel method that estimates the mathematical expectations of rewards at various reasoning steps using tree sampling.<n>Building on the group-relative reward training mechanism of GRPO, name innovatively computes rewards based on step-level groups generated during tree sampling.
  arXiv  Detail & Related papers  (2025-06-05T15:56:38Z)
• Response-Level Rewards Are All You Need for Online Reinforcement Learning in LLMs: A Mathematical Perspective [6.069069082518759]
  We study the Zero-Reward Assumption in reinforcement learning for large language models (LLMs)<n>We show that the policy gradient based on true, unknown token-level rewards can be unbiasedly estimated using only a response-level reward model.<n>We propose a new algorithm: Token-Reinforced Policy Optimization (TRePO)
  arXiv  Detail & Related papers  (2025-06-03T07:44:31Z)
• Reinforced Latent Reasoning for LLM-based Recommendation [92.56166822197919]
  Large Language Models (LLMs) have demonstrated impressive reasoning capabilities in complex problem-solving tasks.<n>Existing methods typically rely on fine-tuning with explicit chain-of-thought (CoT) data.<n>In this work, we explore an alternative approach that shifts from explicit CoT reasoning to compact, information-dense latent reasoning.
  arXiv  Detail & Related papers  (2025-05-25T11:03:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.