Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning

• URL: http://arxiv.org/abs/2606.01682v1
• Date: Mon, 01 Jun 2026 04:43:36 GMT
• Title: Off-the-Shelf LLMs as Process Scorers: Training-Free Alternative to PRMs for Mathematical Reasoning
• Abstract summary: Chunk-Level Guided Generation is a training-free alternative that uses an off-the-shelf large language model as a process scorer.<n>We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias.<n>Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.
• Score: 51.88950852117154
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Selecting the best response from multiple small-model samples using a stronger scorer is a simple inference-time strategy, but fails when the small model has already committed to incorrect reasoning paths. PRM guided search avoids this by scoring candidate continuations during generation, but requires a reward model trained with step-level labels. We propose Chunk-Level Guided Generation, a training-free alternative that uses an off-the-shelf large language model as a process scorer. At each step, a small model samples k fixed-length candidate chunks, while the larger model scores the candidates using likelihoods without generating any text. The selected chunk is committed before the next step, steering generation before errors can propagate. We instantiate this framework with two selection rules: Likelihood-Guided Selection (LGS), which selects the chunk with the highest length-normalized large-model log-probability, and Contrastive-Guided Selection (CGS), which subtracts the small model's log-probability to favor chunks where the large model's preference diverges from the small model's. We show that scoring variable-length reasoning steps with large-model likelihoods is unreliable due to a systematic length bias that persists even after length normalization, and that fixed-length chunks avoid this confound. On GSM8K, MATH, Minerva Math, AMC23, and AIME24 with Qwen2.5-1.5B guided by Qwen2.5-32B and Llama-3.2-1B guided by Llama-3.1-70B, CGS outperforms majority voting by up to 28 pp and, under matched guidance budgets, matches or outperforms Qwen2.5-Math-PRM-72B guided search on most benchmarks without reward-model training. With Qwen2.5-7B guided by Qwen2.5-72B, CGS reaches 81.8% on MATH and 63.6% on Minerva Math at k=16, surpassing majority voting by 4--6 pp. Finally, Chunk-Level Guided Generation produces substantially shorter reasoning traces than PRM guided search.

Related papers

• Models for minimalist RAG: B1ade 335M Embedding and 1B Parameter Small Language Models [2.1665689529884697]
  We present B1ade, an efficient RAG architecture comprising two purpose-built components: a compact embedding model and a purpose-built SLM.<n>B1ade-embed, a 335M parameter retrieval model constructed via parameter-free fusion of five pretrained encoders achieves top MTEB scores among sub-500M models with zero additional training.<n>B1ade-1B cites retrieved passages in 42.4% of responses, exceeding the attribution rate of its training distribution by 5.5 percentage points.
  arXiv  Detail & Related papers  (2026-07-29T22:41:25Z)
• Influence of Prompt Engineering on Small Language Models for Guarded Query Routing [0.0]
  We investigate whether compact open-weight Small Language Models (SLMs) can jointly handle both tasks under latency constraints.<n>We find that mid-scale SLMs come close to frontier model routing quality at much lower latency.<n>Results show that prompt optimization techniques enable SLMs to handle such cases gracefully, without changing the models' weights.
  arXiv  Detail & Related papers  (2026-07-06T09:21:18Z)
• MentorCollab: Selective Large-to-Small Inference-Time Guidance for Efficient Reasoning [85.05204262206296]
  Large reasoning models (LRMs) achieve strong performance by producing long chains of thought, but their inference costs are high.<n>Small language models (SLMs) are far more efficient, yet struggle on multi-step reasoning tasks.<n>We propose MentorCollab, an inference-time collaboration method in which an LRM selectively and sparsely guides an SLM, rather than taking over generation.
  arXiv  Detail & Related papers  (2026-02-05T04:58:16Z)
• Correct, Concise and Complete: Multi-stage Training For Adaptive Reasoning [11.179446105672461]
  We propose a multi-stage efficient reasoning method that combines supervised fine-tuning and reinforcement learning.<n>Our approach reduces response length by an average of 28% for 8B models and 40% for 32B models.<n>It achieves a superior trade-off compared to more complex state-of-the-art efficient reasoning methods.
  arXiv  Detail & Related papers  (2026-01-06T12:31:51Z)
• Error Typing for Smarter Rewards: Improving Process Reward Models with Error-Aware Hierarchical Supervision [20.09181711927194]
  We introduce PathFinder-PRM, a novel hierarchical, error-aware discriminative PRM.<n>On PRMBench, PathFinder-PRM achieves a new state-of-the-art PRMScore of 67.7, outperforming the prior best (65.5) while using 3 times less data.
  arXiv  Detail & Related papers  (2025-05-26T08:56:36Z)
• Value-Guided Search for Efficient Chain-of-Thought Reasoning [49.971608979012366]
  We propose a simple and efficient method for value model training on long-context reasoning traces.<n>By collecting a dataset of 2.5 million reasoning traces, we train a 1.5B token-level value model.<n>We find that block-wise value-guided search (VGS) with a final weighted majority vote achieves better test-time scaling than standard methods.
  arXiv  Detail & Related papers  (2025-05-23T01:05:07Z)
• S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models [2.9925837108958864]
  Test-Time Scaling emerges as an active research focus in the large language model community.<n>Recent studies reveal that reasoning models (even Qwen3) consistently exhibit excessive thought redundancy.<n>This paper introduces Serial-Group Decaying-Reward Policy Optimization (S-GRPO), a novel reinforcement learning paradigm.
  arXiv  Detail & Related papers  (2025-05-12T15:50:44Z)
• Sample, Don't Search: Rethinking Test-Time Alignment for Language Models [55.2480439325792]
  We introduce QAlign, a new test-time alignment approach.<n>As we scale test-time compute, QAlign converges to sampling from the optimal aligned distribution for each individual prompt.<n>By adopting recent advances in Markov chain Monte Carlo for text generation, our method enables better-aligned outputs without modifying the underlying model or even requiring logit access.
  arXiv  Detail & Related papers  (2025-04-04T00:41:40Z)
• Ranked from Within: Ranking Large Multimodal Models Without Labels [73.96543593298426]
  We show that uncertainty scores derived from softmax distributions provide a robust basis for ranking models across various tasks.<n>This facilitates the ranking of LMMs on unlabeled data, providing a practical approach for selecting models for diverse target domains without requiring manual annotation.
  arXiv  Detail & Related papers  (2024-12-09T13:05:43Z)
• Pruning Large Language Models via Accuracy Predictor [0.0]
  Large language models (LLMs) containing tens of billions of parameters (or even more) have demonstrated impressive capabilities in various NLP tasks. We propose a novel pruning approach: firstly, a training set of a certain number of architecture-accuracy pairs is established, and then a non-neural model is trained as an accuracy predictor.
  arXiv  Detail & Related papers  (2023-09-18T06:38:24Z)
• Confidence-Based Model Selection: When to Take Shortcuts for Subpopulation Shifts [119.22672589020394]
  We propose COnfidence-baSed MOdel Selection (CosMoS), where model confidence can effectively guide model selection. We evaluate CosMoS on four datasets with spurious correlations, each with multiple test sets with varying levels of data distribution shift.
  arXiv  Detail & Related papers  (2023-06-19T18:48:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.