From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning

• URL: http://arxiv.org/abs/2606.07190v1
• Date: Fri, 05 Jun 2026 11:56:50 GMT
• Title: From Correctness to Utility: Gain-Based Prefix Evaluation for LLM Reasoning
• Abstract summary: We argue that correctness is a useful but indirect proxy for the effect we ultimately care about.<n>We define this effect as prefix gain, the solve-rate improvement induced by conditioning lightweight student model group on a prefix.<n>We use this to train a Prefix Utility Model (PUM) with a simple pairwise ranking objective.
• Score: 27.89240261179023
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Reasoning prefixes shape the future trajectory of LLM problem solving, yet existing process reward models usually evaluate them through local step correctness. We argue that correctness is a useful but indirect proxy for the effect we ultimately care about: whether a prefix increases the probability of successful completion. We define this effect as prefix gain, the solve-rate improvement induced by conditioning lightweight student model group on a prefix, and use it to train a Prefix Utility Model (PUM) with a simple pairwise ranking objective. PUM learns outcome-grounded prefix utility and can score both complete trajectories and partial reasoning prefixes. Across Best-of-$N$ selection, beam search, and reinforcement learning on mathematical reasoning, PUM provides a strong prefix-level supervision signal, especially when candidate pools are large, search budgets increase, or rule-based rewards are sparse. We release all data, models, and code at https://zhiqix.github.io/pum-project-page.

Related papers

• When and Why LLM Causal Priors Help: Closed-Loop Prior Selection for Amortized Causal Inference [0.0]
  Causal effect estimation asks how an outcome would change under an intervention, medicine, economics, and public policy all treat it as a task.<n>Large language models (LLMs) can now draw'' plausible causal graphs for a given domain.<n>We propose a emphclosed-loop prior selection framework that casts prior injection as a budget-constrained optimization over a candidate prior pool.
  arXiv  Detail & Related papers  (2026-09-07T02:24:11Z)
• Prefix-Guided On-Policy Distillation: Mining Golden Trajectories from Rollouts [48.550535291129584]
  Prefix-Guided On-Policy Distillation (PG-OPD) is a simple rollout-allocation framework that uses fixed-length prefixes to estimate trajectory value before expensive long-horizon generation.<n>Across diverse teacher-student combinations on AMC, AIME, and HMMT benchmarks, PG-OPD improves average accuracy by up to 4.80 points while reducing training time by up to 2.46x.
  arXiv  Detail & Related papers  (2026-06-20T11:18:34Z)
• PriFT: Prior-Support Guided Supervised Fine-Tuning [74.65198014829393]
  Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation.<n>SFT fits fixed demonstrations token by token, including targets poorly aligned with the model's pretrained distribution.<n>We propose PriFT, which derives token weights from a frozen pretrained reference to obtain a stable reweighting signal unaffected by fine-tuning.
  arXiv  Detail & Related papers  (2026-06-08T12:14:06Z)
• $\nabla$-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space [71.23672814629448]
  $nabla$-Reasoner is an iterative generation framework that integrates differentiable optimization over token logits into the decoding loop.<n>$nabla$-Reasoner achieves over 20% accuracy improvement on a challenging mathematical reasoning benchmark.
  arXiv  Detail & Related papers  (2026-03-05T08:42:54Z)
• APAO: Adaptive Prefix-Aware Optimization for Generative Recommendation [26.371939617653084]
  Generative recommendation is an autoregressive generation process, predicting discrete tokens of the next item conditioned on user interaction histories.<n>Existing generative recommendation models are typically trained with token-level likelihood objectives, such as cross-entropy loss.<n>This leads to a training-inference inconsistency: standard training assumes ground-truth history is always available, ignoring the fact that beam search prunes low-probability branches during inference.
  arXiv  Detail & Related papers  (2026-03-03T08:29:15Z)
• Save the Good Prefix: Precise Error Penalization via Process-Supervised RL to Enhance LLM Reasoning [59.76691952347156]
  Reinforcement learning (RL) has emerged as a powerful framework for improving the reasoning capabilities of large language models (LLMs)<n>Most existing RL approaches rely on sparse outcome rewards, which fail to credit correct intermediate steps in partially successful solutions.<n>We propose Verifiable Prefix Policy Optimization (VPPO), which uses PRMs only to localize the first error during RL.
  arXiv  Detail & Related papers  (2026-01-26T21:38:20Z)
• Knowing the Answer Isn't Enough: Fixing Reasoning Path Failures in LVLMs [85.37131922131657]
  We reveal a critical yet underexplored flaw in Large Vision-Language Models (LVLMs)<n>Even when these models know the correct answer, they frequently arrive there through incorrect reasoning paths.<n>We propose PSO (Path-Select Optimization), a two-stage post-training framework designed to enhance both the reasoning performance and stability of existing LVLMs.
  arXiv  Detail & Related papers  (2025-12-06T03:02:55Z)
• FlowRL: Matching Reward Distributions for LLM Reasoning [69.88820066093798]
  We propose FlowRL: matching the full reward distribution via flow balancing instead of maximizing rewards in large language model (LLM) reinforcement learning (RL)<n>We transform scalar rewards into a normalized target distribution using a learnable partition function, and then minimize the reverse KL divergence between the policy and the target distribution.
  arXiv  Detail & Related papers  (2025-09-18T17:56:36Z)
• Accelerating LLM Reasoning via Early Rejection with Partial Reward Modeling [12.835376812101323]
  We introduce the hypothesis that PRMs are also Partial Reward Models.<n>This allows for principled early rejection based on intermediate token-level signals.<n>On math reasoning benchmarks, our method achieves up to 1.4$times$-9$times$ reduction in inference FLOPs without degrading final performance.
  arXiv  Detail & Related papers  (2025-08-04T00:58:56Z)
• The First Few Tokens Are All You Need: An Efficient and Effective Unsupervised Prefix Fine-Tuning Method for Reasoning Models [69.798277882245]
  We introduce Unsupervised Prefix Fine-Tuning (UPFT) to enhance large language models' reasoning efficiency.<n>UPFT removes the need for labeled data or exhaustive sampling.<n> Experiments show that UPFT matches the performance of supervised methods.
  arXiv  Detail & Related papers  (2025-03-04T18:56:03Z)
• Online Preference Alignment for Language Models via Count-based Exploration [46.46627519343809]
  Reinforcement Learning from Human Feedback (RLHF) has shown great potential in fine-tuning Large Language Models (LLMs) to align with human preferences.<n>Existing methods perform preference alignment from a fixed dataset, which can be limited in data coverage.<n>Online RLHF is more desirable to empower the LLM to explore outside the support of the initial dataset by iteratively collecting the prompt-response pairs.
  arXiv  Detail & Related papers  (2025-01-22T09:12:09Z)
• FIRST: Faster Improved Listwise Reranking with Single Token Decoding [56.727761901751194]
  First, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.
  arXiv  Detail & Related papers  (2024-06-21T21:27:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.