Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling

• URL: http://arxiv.org/abs/2510.05825v1
• Date: Tue, 07 Oct 2025 11:48:32 GMT
• Title: Mitigating Premature Exploitation in Particle-based Monte Carlo for Inference-Time Scaling
• Abstract summary: Inference-Time Scaling (ITS) improves language models by allocating more computation at generation time.<n>PF has emerged as a strong ITS method for complex mathematical reasoning tasks.<n>It is vulnerable when guided by process reward models, which often assign overconfident scores early in the reasoning process.<n>This failure mode, known as particle impoverishment, is especially severe under constrained computational budgets.<n>We introduce Entropic Particle Filtering (ePF), an algorithm that integrates two new techniques to solve these issues.
• Score: 15.828750560145751
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Inference-Time Scaling (ITS) improves language models by allocating more computation at generation time. Particle Filtering (PF) has emerged as a strong ITS method for complex mathematical reasoning tasks, but it is vulnerable when guided by process reward models, which often assign overconfident scores early in the reasoning process. This causes PF to suffer from premature exploitation: it myopically commits to locally promising trajectories, prunes potentially correct hypotheses, and converges to suboptimal solutions. This failure mode, known as particle impoverishment, is especially severe under constrained computational budgets. To address this, we analyze the problem and identify two root causes: a lack of diversity in the particle set due to overconfident resampling and consequent inability to assess the potential of a reasoning path. We introduce Entropic Particle Filtering (ePF), an algorithm that integrates two new techniques to solve these issues. The first technique, Entropic Annealing (EA), directly mitigates particle impoverishment by monitoring search diversity via entropy; when diversity drops, it intervenes by dynamically annealing the resampling distribution to preserve exploration. The second, an enhancement called Look-ahead Modulation (LaM), adds a predictive guide to evaluate a state's potential based on its successors. On several challenging math benchmarks, ePF significantly outperforms strong baselines and achieves up to a 50 % relative improvement in task reward. Together, these methods improve PF's resilience by balancing the exploration of diverse solution spaces with the exploitation of high-reward regions, ultimately leading to higher-quality solutions.

Related papers

• Generative Model Proposal based Particle Filtering for Data Assimilation [13.045651122066927]
  Flow Proposal Particle Filters learn a conditional generative model based proposal approximating the variance-minimizing optimal proposal for particle propagation.<n>Since our proposal admits tractable likelihood evaluation, FPPF computes accurate importance weights and retains a Bayesian update step.
  arXiv  Detail & Related papers  (2026-07-01T14:48:32Z)
• Intrinsic Selection and Particle Resampling for Inference-Time Scaling Beyond Domain Verifiability [15.77992479706947]
  Inference-Time Scaling (ITS) has largely succeeded in verifiable domains like math and coding.<n>However, extending ITS to tasks prone to systematic failure typically relies on costly external solvers or brittle, model-based verifiers.<n>Our pipeline applies seamlessly across broad-purpose, domain-specialized, and multimodal architectures.
  arXiv  Detail & Related papers  (2026-06-07T21:43:37Z)
• Projected Inverse Iteration: An Eigenvalue Approach to Ground-State Computation with Neural Quantum States [17.26784290982439]
  Deep learning offers a powerful approach to quantum many-body problems via neural network wavefunctions.<n>Existing optimization methods, including natural descent and gradient reconfiguration, suffer from spectral gap-dependent convergence.<n>Here, we introduce Projected Inverse Iteration (PII) by re-framing the ground-state search as an eigenvalue problem.
  arXiv  Detail & Related papers  (2026-06-05T20:23:38Z)
• Parallel Diffusion Solver via Residual Dirichlet Policy Optimization [88.7827307535107]
  Diffusion models (DMs) have achieved state-of-the-art generative performance but suffer from high sampling latency due to their sequential denoising nature.<n>Existing solver-based acceleration methods often face significant image quality degradation under a low-dimensional budget.<n>We propose the Ensemble Parallel Direction solver (dubbed as EPD-EPr), a novel ODE solver that mitigates these errors by incorporating multiple gradient parallel evaluations in each step.
  arXiv  Detail & Related papers  (2025-12-28T05:48:55Z)
• Adaptive Probability Flow Residual Minimization for High-Dimensional Fokker-Planck Equations [14.22534820071447]
  Solving high-dimensional Fokker-Planck equations is a challenge in computational physics and dynamics.<n>Existing deep learning approaches, such as Physics-Informed Neural Networks, face computational challenges as dimensionality increases.<n>We propose the Adaptive Probability Flow Residual Minimization (A-PFRM) method.
  arXiv  Detail & Related papers  (2025-12-22T09:31:31Z)
• DaGRPO: Rectifying Gradient Conflict in Reasoning via Distinctiveness-Aware Group Relative Policy Optimization [20.66452395111739]
  We propose Distinctiveness-aware Group Relative Policy Optimization (DaGRPO)<n>DaGRPO incorporates two core mechanisms: (1) Sequence-level Gradient Rectification, which utilizes fine-grained scoring to dynamically mask sample pairs with low distinctiveness; and (2) Off-policy Data Augmentation, which introduces high-quality anchors to recover training signals for challenging tasks.<n>In-depth analysis confirms that DaGRPO effectively mitigates gradient explosion and accelerates the emergence of long-chain reasoning capabilities.
  arXiv  Detail & Related papers  (2025-12-06T07:51:36Z)
• High-dimensional Mean-Field Games by Particle-based Flow Matching [18.129646808071893]
  Mean-field games (MFGs) study the Nash equilibrium of systems with a continuum of interacting agents.<n>Despite their broad applicability, solving high-dimensional MFGs remains a significant challenge due to fundamental computational and analytical obstacles.<n>We propose a particle-based deep Flow Matching (FM) method to tackle high-dimensional MFGs.
  arXiv  Detail & Related papers  (2025-12-01T01:04:53Z)
• Enhancing Adversarial Transferability by Balancing Exploration and Exploitation with Gradient-Guided Sampling [82.52485740425321]
  Adrial attacks present a critical challenge to deep neural networks' robustness.<n> transferability of adversarial attacks faces a dilemma between Exploitation (maximizing attack potency) and Exploration (enhancing cross-model generalization)
  arXiv  Detail & Related papers  (2025-11-01T05:43:47Z)
• SynCast: Synergizing Contradictions in Precipitation Nowcasting via Diffusion Sequential Preference Optimization [62.958457694151384]
  We introduce preference optimization into precipitation nowcasting for the first time, motivated by the success of reinforcement learning from human feedback in large language models.<n>In the first stage, the framework focuses on reducing FAR, training the model to effectively suppress false alarms.
  arXiv  Detail & Related papers  (2025-10-22T16:11:22Z)
• Improving monotonic optimization in heterogeneous multi-agent reinforcement learning with optimal marginal deterministic policy gradient [18.64288030584699]
  heterogeneous multi-agent reinforcement learning (MARL)<n>Objectively replace the sequentially computed $Q_psi*(s,a_1:i)$ with the Optimal Marginal Q function $phi_psi*(s,a_1:i)$ derived from Q-functions.<n>Generalized Q Critic (GQC) as the critic function, employing pessimistic uncertainty-constrained loss to optimize different Q-value estimations.
  arXiv  Detail & Related papers  (2025-07-14T07:16:01Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Pseudospectral method for solving PDEs using Matrix Product States [0.0]
  This research focuses on solving time-dependent partial differential equations (PDEs) using matrix product states (MPS)<n>We propose an extension of Hermite Distributed Approximating Functionals (HDAF) to MPS, a highly accurate pseudospectral method for approximating functions of derivatives.
  arXiv  Detail & Related papers  (2024-09-04T17:53:38Z)
• Persistent Sampling: Enhancing the Efficiency of Sequential Monte Carlo [0.0]
  Sequential Monte Carlo (SMC) samplers are powerful tools for Bayesian inference but suffer from high computational costs.<n>We introduce persistent sampling (PS), which retains SMC and constructs particles from all prior iterations.
  arXiv  Detail & Related papers  (2024-07-30T10:34:40Z)
• Non-convex Bayesian Learning via Stochastic Gradient Markov Chain Monte Carlo [4.656426393230839]
  The rise of artificial intelligence (AI) hinges on efficient of modern deep neural networks (DNNs) for non-trips and uncertainty. In this thesis we propose a tool to handle the problem of Monte Carlo exploitation. We also propose two dynamic importance sampling algorithms for the underlying ordinary equation (ODE) system.
  arXiv  Detail & Related papers  (2023-05-30T18:25:11Z)
• PAPAL: A Provable PArticle-based Primal-Dual ALgorithm for Mixed Nash Equilibrium [58.26573117273626]
  We consider the non-AL equilibrium nonconptotic objective function in two-player zero-sum continuous games. Our novel insights into the particle-based algorithms for continuous distribution strategies are presented.
  arXiv  Detail & Related papers  (2023-03-02T05:08:15Z)
• Harnessing Heterogeneity: Learning from Decomposed Feedback in Bayesian Modeling [68.69431580852535]
  We introduce a novel GP regression to incorporate the subgroup feedback. Our modified regression has provably lower variance -- and thus a more accurate posterior -- compared to previous approaches. We execute our algorithm on two disparate social problems.
  arXiv  Detail & Related papers  (2021-07-07T03:57:22Z)
• Second-Order Guarantees in Centralized, Federated and Decentralized Nonconvex Optimization [64.26238893241322]
  Simple algorithms have been shown to lead to good empirical results in many contexts. Several works have pursued rigorous analytical justification for studying non optimization problems. A key insight in these analyses is that perturbations play a critical role in allowing local descent algorithms.
  arXiv  Detail & Related papers  (2020-03-31T16:54:22Z)
• Targeted free energy estimation via learned mappings [66.20146549150475]
  Free energy perturbation (FEP) was proposed by Zwanzig more than six decades ago as a method to estimate free energy differences. FEP suffers from a severe limitation: the requirement of sufficient overlap between distributions. One strategy to mitigate this problem, called Targeted Free Energy Perturbation, uses a high-dimensional mapping in configuration space to increase overlap.
  arXiv  Detail & Related papers  (2020-02-12T11:10:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.