PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament

• URL: http://arxiv.org/abs/2501.13007v2
• Date: Wed, 19 Feb 2025 13:35:48 GMT
• Title: PairJudge RM: Perform Best-of-N Sampling with Knockout Tournament
• Abstract summary: Pairwise Judge Reward Model (PariJudge RM) combined with a knockout tournament for BoN sampling.<n>Instead of assigning absolute scores, PariJudge RM judges two candidate solutions' correctness with chain-of-thought reasoning simultaneously.<n>In the knockout tournament, PariJudge RM conducts pairwise Judgment between candidate solutions and eliminates the incorrect ones iteratively.
• Score: 37.97757796124621
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Best-of-N (BoN) sampling, a common strategy for test-time scaling of Large Language Models (LLMs), relies on reward models to select the best candidate solution from multiple generations. However, traditional reward models often assign arbitrary and inconsistent scores, limiting their effectiveness. To address this, we propose a Pairwise Judge Reward Model (PariJudge RM) combined with a knockout tournament for BoN sampling. Instead of assigning absolute scores, given one math problem, PariJudge RM judges two candidate solutions' correctness with chain-of-thought reasoning simultaneously. This approach eliminates the need for scoring and enables cross-validation of solutions through parallel judgment. In the knockout tournament, PariJudge RM conducts pairwise Judgment between candidate solutions and eliminates the incorrect ones iteratively. We construct PairJudge-432K, a large-scale dataset of 432K pairwise judgments derived from NumiaMath and annotated using \texttt{gemini-1.5-flash}, and train the PariJudge RM via supervised fine-tuning. Experiments on MATH-500 and the Olympiad Bench demonstrate significant improvements over baseline reward models. And a 40\% to 60\% relative improvement is achieved on the top 50\% challenging problems.

Related papers

• Refining Over Resampling: Test-Time Self-Correction for LLM Reasoning [3.8191765711200065]
  We propose a verifier-free breadth--depth refinement framework that uses test-time compute to both explore and improve candidate solutions.<n>Our method consistently improves over greedy decoding, majority voting, verifier-based best-of-$N$, beam search, and look decoding across multiple open-weight models.
  arXiv  Detail & Related papers  (2026-08-06T06:38:37Z)
• Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning [51.88950852117154]
  Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
  arXiv  Detail & Related papers  (2026-06-01T04:43:36Z)
• MARBLE: Multi-Aspect Reward Balance for Diffusion RL [71.6241143519038]
  Reinforcement learning has become the dominant approach for aligning diffusion models with human preferences.<n>Existing practice deal with multiple rewards by training one specialist model per reward.<n>We propose MARBLE, a gradient-space optimization framework that maintains independent advantage estimators for each reward.
  arXiv  Detail & Related papers  (2026-05-07T16:20:42Z)
• Test-Time Scaling with Diffusion Language Models via Reward-Guided Stitching [66.39914384073145]
  We propose a self-consistency framework that turns cheap diffusion-sampled reasoning into a reusable pool of step-level candidates.<n>We find that step-level recombination is most beneficial on harder problems.<n>Our training-free framework improves average accuracy by up to 2 across six math and coding tasks.
  arXiv  Detail & Related papers  (2026-02-26T11:08:39Z)
• Sample Smart, Not Hard: Correctness-First Decoding for Better Reasoning in LLMs [72.82403830490084]
  We argue that the decoding rule should be calibrated by correctness, not confidence alone.<n>We propose simple strategies that achieve this goal: Greedy-Threshold makes sampling greedy at very low confidence steps.<n>Together, our findings challenge prevailings about decoding under uncertainty and show gains across math and general reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-10-07T14:46:12Z)
• GuidedSampling: Steering LLMs Towards Diverse Candidate Solutions at Inference-Time [40.94165318043027]
  We propose a new inference algorithm, GuidedSampling, which decouples the exploration and generation phases during inference.<n>The exploration phase identifies multiple concepts that can be utilized to solve the problem, while the generation phase applies a specific concept to provide final solution candidates.
  arXiv  Detail & Related papers  (2025-10-04T11:02:39Z)
• RIMO: An Easy-to-Evaluate, Hard-to-Solve Olympiad Benchmark for Advanced Mathematical Reasoning [26.173204350710833]
  RIMO is a two-track benchmark designed to preserve peak Olympiad difficulty while eliminating evaluation noise.<n>The first track, RIMO-N, rewrites 335 problems to admit a single, unique integer answer, allowing for deterministic correctness checking.<n>The second track, RIMO-P, features 456 proof problems with expert-checked solutions, which are decomposed into a sequence of sub-problems to evaluate the step-by-step reasoning process.
  arXiv  Detail & Related papers  (2025-09-09T13:13:51Z)
• Batched Stochastic Matching Bandits [43.651070266360954]
  We introduce a novel bandit framework for matching based on the Multi-nomial Logit (MNL) choice model.<n>In our setting, $N$ agents on one side are assigned to $K$ arms on the other side, where each armally selects an agent from its assigned pool according to an unknown preference.<n>The objective is to minimize regret by maximizing the cumulative revenue from successful matches across all agents.
  arXiv  Detail & Related papers  (2025-09-04T13:16:32Z)
• PiCSAR: Probabilistic Confidence Selection And Ranking [46.51719514052471]
  Key challenge for reasoning tasks is designing a scoring function that can identify correct reasoning chains without access to ground-truth answers.<n>We propose Probabilistic Confidence Selection And Ranking (PiCSAR), a training-free method that scores each candidate generation using the joint log-likelihood of the reasoning and final answer.<n>Our analysis reveals that correct reasoning chains exhibit significantly higher reasoning and answer confidence, justifying the effectiveness of PiCSAR.
  arXiv  Detail & Related papers  (2025-08-29T17:03:47Z)
• Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision [20.09181711927194]
  We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM.<n>On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3 times less data.
  arXiv  Detail & Related papers  (2025-05-26T08:56:36Z)
• LookAlike: Consistent Distractor Generation in Math MCQs [42.19039301965107]
  We propose LookAlike, a method that improves error-distractor consistency via preference optimization.<n>Our two main innovations are: (a) mining synthetic preference pairs from model inconsistencies, and (b) alternating supervised fine-tuning.<n>LookAlike achieves 51.6% accuracy in distractor generation and 57.2% in error generation under LLM-as-a-judge evaluation.
  arXiv  Detail & Related papers  (2025-05-03T19:18:06Z)
• Sample, Don't Search: Rethinking Test-Time Alignment for Language Models [55.2480439325792]
  We introduce QAlign, a new test-time alignment approach. As we scale test-time compute, QAlign converges to sampling from the optimal aligned distribution for each individual prompt. By adopting recent advances in Markov chain Monte Carlo for text generation, our method enables better-aligned outputs without modifying the underlying model or even requiring logit access.
  arXiv  Detail & Related papers  (2025-04-04T00:41:40Z)
• LLaMA-Berry: Pairwise Optimization for O1-like Olympiad-Level Mathematical Reasoning [56.273799410256075]
  The framework combines Monte Carlo Tree Search (MCTS) with iterative Self-Refine to optimize the reasoning path. The framework has been tested on general and advanced benchmarks, showing superior performance in terms of search efficiency and problem-solving capability.
  arXiv  Detail & Related papers  (2024-10-03T18:12:29Z)
• Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis [16.288866201806382]
  We develop a model-free RLHF best policy identification algorithm, called $mathsfBSAD$, without explicit reward model inference.<n>The algorithm identifies the optimal policy directly from human preference information in a backward manner.
  arXiv  Detail & Related papers  (2024-06-11T17:01:41Z)
• Best Arm Identification with Fixed Budget: A Large Deviation Perspective [54.305323903582845]
  We present sred, a truly adaptive algorithm that can reject arms in it any round based on the observed empirical gaps between the rewards of various arms. In particular, we present sred, a truly adaptive algorithm that can reject arms in it any round based on the observed empirical gaps between the rewards of various arms.
  arXiv  Detail & Related papers  (2023-12-19T13:17:43Z)
• AffineGlue: Joint Matching and Robust Estimation [74.04609046690913]
  We propose AffineGlue, a method for joint two-view feature matching and robust estimation. AffineGlue selects potential matches from one-to-many correspondences to estimate minimal models. Guided matching is then used to find matches consistent with the model, suffering less from the ambiguities of one-to-one matches.
  arXiv  Detail & Related papers  (2023-07-28T08:05:36Z)
• Fit Like You Sample: Sample-Efficient Generalized Score Matching from Fast Mixing Diffusions [29.488555741982015]
  We show a close connection between the mixing time of a broad class of Markov processes with generator $mathcalL$. We adapt techniques to speed up Markov chains to construct better score-matching losses. In particular, preconditioning' the diffusion can be translated to an appropriate preconditioning'' of the score loss.
  arXiv  Detail & Related papers  (2023-06-15T17:58:42Z)
• Implicitly normalized forecaster with clipping for linear and non-linear heavy-tailed multi-armed bandits [85.27420062094086]
  Implicitly Normalized Forecaster (INF) is considered an optimal solution for adversarial multi-armed bandit (MAB) problems. We propose a new version of INF called the Implicitly Normalized Forecaster with clipping (INFclip) for MAB problems with heavy-tailed settings. We demonstrate that INFclip is optimal for linear heavy-tailed MAB problems and works well for non-linear ones.
  arXiv  Detail & Related papers  (2023-05-11T12:00:43Z)
• Two-sided Competing Matching Recommendation Markets With Quota and Complementary Preferences Constraints [13.069703665055084]
  We propose a new recommendation algorithm for addressing the problem of two-sided online matching markets with complementary preferences and quota constraints. The presence of mixed quota and complementary preferences constraints can lead to instability in the matching process. We formulate the problem as a bandit learning framework and propose the Multi-agent Multi-type Thompson Sampling algorithm.
  arXiv  Detail & Related papers  (2023-01-24T18:54:29Z)
• Enhancing Self-Consistency and Performance of Pre-Trained Language Models through Natural Language Inference [72.61732440246954]
  Large pre-trained language models often lack logical consistency across test inputs. We propose a framework, ConCoRD, for boosting the consistency and accuracy of pre-trained NLP models. We show that ConCoRD consistently boosts accuracy and consistency of off-the-shelf closed-book QA and VQA models.
  arXiv  Detail & Related papers  (2022-11-21T21:58:30Z)
• Reducing Variance in Temporal-Difference Value Estimation via Ensemble of Deep Networks [109.59988683444986]
  MeanQ is a simple ensemble method that estimates target values as ensemble means. We show that MeanQ shows remarkable sample efficiency in experiments on the Atari Learning Environment benchmark.
  arXiv  Detail & Related papers  (2022-09-16T01:47:36Z)
• Deep Probabilistic Graph Matching [72.6690550634166]
  We propose a deep learning-based graph matching framework that works for the original QAP without compromising on the matching constraints. The proposed method is evaluated on three popularly tested benchmarks (Pascal VOC, Willow Object and SPair-71k) and it outperforms all previous state-of-the-arts on all benchmarks.
  arXiv  Detail & Related papers  (2022-01-05T13:37:27Z)
• Bootstrap Your Object Detector via Mixed Training [82.98619147880397]
  MixTraining is a new training paradigm for object detection that can improve the performance of existing detectors for free. It enhances data augmentation by utilizing augmentations of different strengths while excluding the strong augmentations of certain training samples that may be detrimental to training. MixTraining is found to bring consistent improvements across various detectors on the COCO dataset.
  arXiv  Detail & Related papers  (2021-11-04T17:58:26Z)
• Exploiting Transitivity for Top-k Selection with Score-Based Dueling Bandits [0.0]
  We consider the problem of top-k subset selection in Dueling Bandit problems with score information. We propose a Thurstonian style model and adapt the Pairwise Optimal Computing Budget Allocation for subset selection (POCBAm) sampling method.
  arXiv  Detail & Related papers  (2020-12-31T14:54:25Z)
• Addressing Class-Imbalance Problem in Personalized Ranking [47.11372043636176]
  We propose an efficient emphunderlineVital underlineNegative underlineSampler (VINS) to alleviate the class-imbalance issue for pairwise ranking model. VINS is a bias sampler with reject probability that will tend to accept a negative candidate with a larger degree weight than the given positive item.
  arXiv  Detail & Related papers  (2020-05-19T08:11:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.