Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo

• URL: http://arxiv.org/abs/2604.16453v1
• Date: Tue, 07 Apr 2026 21:48:04 GMT
• Title: Sampling for Quality: Training-Free Reward-Guided LLM Decoding via Sequential Monte Carlo
• Abstract summary: We introduce a principled probabilistic framework for reward-guided decoding in large language models.<n>Our method defines a reward-augmented target distribution over complete sequences.<n>It modifies the inference distribution via reward potentials, with all gains arising purely from inference-time sampling.
• Score: 16.197322348431516
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a principled probabilistic framework for reward-guided decoding in large language models, addressing the limitations of standard decoding methods that optimize token-level likelihood rather than sequence-level quality. Our method defines a reward-augmented target distribution over complete sequences by combining model transition probabilities with prefix-dependent reward potentials. Importantly, the approach is training-free: it leaves model weights unchanged and instead modifies the inference distribution via reward potentials, with all gains arising purely from inference-time sampling. To sample from this distribution, we develop Sequential Monte Carlo algorithms, including a computationally efficient prefix-only variant and a lookahead variant whose intermediate targets match the exact marginals of the full sequence distribution. The framework also integrates resample-move updates with Metropolis-Hastings rejuvenation and supports block-wise generation, subsuming common decoding strategies such as temperature sampling and power-tempered objectives. Empirical results across three 7B models show significant gains. On code generation (HumanEval), our method improves base performance by up to 54.9% and surpasses the strongest sampling baselines by 9.1%-15.3%. On mathematical reasoning (MATH500), it achieves gains of up to 8.8%. Notably, it reaches 87.8% on HumanEval and 78.4% on MATH500 with Qwen2.5-7B, consistently outperforming the reinforcement learning method GRPO.

Related papers

• Optimizing Visual Generative Models via Distribution-wise Rewards [57.46109819595665]
  We present a novel framework that finetunes generative models using distribution-wise rewards, ensuring better alignment with real-world data distributions.<n>Our approach significantly improves FID-50K across various base models, from 5.77 for SiT and from 3.74 to 3.52 for EDM2.
  arXiv  Detail & Related papers  (2026-07-02T15:08:56Z)
• Learning Ordinal Probabilistic Reward from Preferences [25.069054134899744]
  We introduce a novel reward modeling paradigm: Probabilistic Reward Model (PRM)<n>Instead of modeling reward as a deterministic scalar, our approach treats it as a random variable, learning a full probability distribution for the quality of each response.<n>Building on OPRM, we propose a data-efficient training strategy called Region Flooding Tuning (RgFT)
  arXiv  Detail & Related papers  (2026-02-13T06:43:02Z)
• Self-Rewarding Sequential Monte Carlo for Masked Diffusion Language Models [58.946955321428845]
  This work presents self-rewarding sequential Monte Carlo (SMC)<n>Our algorithm stems from the observation that most existing MDLMs rely on a confidence-based sampling strategy.<n>We introduce the trajectory-level confidence as a self-rewarding signal for assigning particle importance weights.
  arXiv  Detail & Related papers  (2026-02-02T09:21:45Z)
• From Static to Dynamic: Adaptive Monte Carlo Search for Mathematical Process Supervision [49.59309446816251]
  Existing methods estimate the quality of reasoning steps based on a fixed-budget sampling strategy.<n>We propose Adaptive Monte Carlo Search (AMCS), a framework that transforms data generation from fixed, static to adaptive.<n>AMCS adaptively refines estimation by allocating more samples to uncertain reasoning steps while using fewer samples for those that are easier to estimate.
  arXiv  Detail & Related papers  (2025-09-29T06:52:35Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• TreeRPO: Tree Relative Policy Optimization [65.51935468270916]
  name is a novel method that estimates the mathematical expectations of rewards at various reasoning steps using tree sampling.<n>Building on the group-relative reward training mechanism of GRPO, name innovatively computes rewards based on step-level groups generated during tree sampling.
  arXiv  Detail & Related papers  (2025-06-05T15:56:38Z)
• Adaptive Inference-Time Compute: LLMs Can Predict if They Can Do Better, Even Mid-Generation [51.127054971591924]
  We introduce a new generative self-evaluation scheme designed to adaptively reduce the number of generated samples. We demonstrate that 74% of the improvement from using 16 samples can be achieved with only 1.2 samples on average.
  arXiv  Detail & Related papers  (2024-10-03T17:47:29Z)
• iBRF: Improved Balanced Random Forest Classifier [0.0]
  Class imbalance poses a major challenge in different classification tasks. We propose a modification to the Balanced Random Forest (BRF) classifier to enhance the prediction performance. Our proposed hybrid sampling technique, when incorporated into the framework of the Random Forest classifier, achieves better prediction performance than other sampling techniques used in imbalanced classification tasks.
  arXiv  Detail & Related papers  (2024-03-14T20:59:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.