Depth-Entropy Guided Sampling for Training-Free LLM Reasoning

• URL: http://arxiv.org/abs/2607.09693v1
• Date: Fri, 19 Jun 2026 12:36:23 GMT
• Title: Depth-Entropy Guided Sampling for Training-Free LLM Reasoning
• Abstract summary: Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models.<n>We introduce Depth-Entropy Guided Sampling (DEGS), a training-free, test-time method that exploits layer-wise entropy collapse as an intrinsic quality signal.
• Score: 7.057431471793534
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Reinforcement learning (RL) has become the dominant paradigm for improving the reasoning capabilities of large language models, but it requires expensive training, curated data, and reward signals. Recent work shows that sampling from sharpened base-model distributions at test time recovers much of the RL gain, yet existing methods rely solely on output-layer likelihoods and ignore the transformer's internal forward-pass dynamics. We introduce Depth-Entropy Guided Sampling (DEGS), a training-free, test-time method that exploits layer-wise entropy collapse as an intrinsic quality signal. We observe that stronger reasoners -- including RL-posttrained variants -- exhibit a distinctive "late collapse": logit-lens decoded entropy stays elevated until deeper layers before converging. We define a per-sequence collapse depth $D(\mathbf{x})$ and a joint objective $π(\mathbf{x}) \propto p(\mathbf{x})^α\exp(βD(\mathbf{x}))$ that combines sequence likelihood with this depth-entropy structure, instantiated inside an MCMC power-sampling framework (DEGS-MCMC). Across three open-weight models and four reasoning benchmarks, this near-chance per-candidate signal compounds over the sampling trajectory into state-of-the-art training-free accuracy, with gains largest out of domain and on the harder splits -- exactly where likelihood alone falls short -- at single-digit-percent wall-clock overhead. DEGS narrowly trails an in-house GRPO reference on the math splits GRPO was trained for, yet surpasses it out of domain on GPQA for all three models, without any training, reward model, or labeled data.

Related papers

• Sampling via Decision-Flow: Training-Free Extraction of Improved Latent Reasoning Paths in Large Language Models [1.8231916504534451]
  Decision-Flow Sampling (DF-Sample) is a training-free, data-free inference-time framework.<n>DF-Sample consistently outperforms baselines across three models and four benchmarks.
  arXiv  Detail & Related papers  (2026-09-11T00:48:45Z)
• PriFT: Prior-Support Guided Supervised Fine-Tuning [74.65198014829393]
  Supervised fine-tuning (SFT) is an efficient approach for downstream task adaptation.<n>SFT fits fixed demonstrations token by token, including targets poorly aligned with the model's pretrained distribution.<n>We propose PriFT, which derives token weights from a frozen pretrained reference to obtain a stable reweighting signal unaffected by fine-tuning.
  arXiv  Detail & Related papers  (2026-06-08T12:14:06Z)
• Posterior Augmented Flow Matching [64.1559809786948]
  Posterior-Augmented Flow Matching (PAFM) is a generalization of flow matching (FM)<n>We show that PAFM improves over FM by up to 3.4 FID50K across different model scales.
  arXiv  Detail & Related papers  (2026-05-01T17:59:59Z)
• Reasoning with Sampling: Your Base Model is Smarter Than You Think [52.639108524651846]
  We propose a simple iterative sampling algorithm leveraging the base models' own likelihoods.<n>We show that our algorithm offers substantial boosts in reasoning that nearly match and even outperform those from RL.<n>Our method does not require training, curated datasets, or a verifier.
  arXiv  Detail & Related papers  (2025-10-16T17:18:11Z)
• Tracing the Representation Geometry of Language Models from Pretraining to Post-training [22.18942718274405]
  We take a spectral approach to investigate the geometry of learned representations across pretraining and post-training.<n>We uncover a consistent non-monotonic sequence of three geometric phases during autoregressive pretraining.<n>Post-training further transforms geometry: SFT and DPO drive "entropy-seeking" dynamics to integrate specific instructional or preferential data.
  arXiv  Detail & Related papers  (2025-09-27T00:46:29Z)
• Inference-Time Scaling of Diffusion Language Models with Particle Gibbs Sampling [70.8832906871441]
  We study how to steer generation toward desired rewards without retraining the models.<n>Prior methods typically resample or filter within a single denoising trajectory, optimizing rewards step-by-step without trajectory-level refinement.<n>We introduce particle Gibbs sampling for diffusion language models (PG-DLM), a novel inference-time algorithm enabling trajectory-level refinement while preserving generation perplexity.
  arXiv  Detail & Related papers  (2025-07-11T08:00:47Z)
• Fractured Chain-of-Thought Reasoning [61.647243580650446]
  We introduce Fractured Sampling, a unified inference-time strategy that interpolates between full CoT and solution-only sampling.<n>We show that Fractured Sampling consistently achieves superior accuracy-cost trade-offs, yielding steep log-linear scaling gains in Pass@k versus token budget.
  arXiv  Detail & Related papers  (2025-05-19T11:30:41Z)
• Reinforcement Learning from Human Feedback without Reward Inference: Model-Free Algorithm and Instance-Dependent Analysis [16.288866201806382]
  We develop a model-free RLHF best policy identification algorithm, called $mathsfBSAD$, without explicit reward model inference.<n>The algorithm identifies the optimal policy directly from human preference information in a backward manner.
  arXiv  Detail & Related papers  (2024-06-11T17:01:41Z)
• Posterior Coreset Construction with Kernelized Stein Discrepancy for Model-Based Reinforcement Learning [78.30395044401321]
  We develop a novel model-based approach to reinforcement learning (MBRL) It relaxes the assumptions on the target transition model to belong to a generic family of mixture models. It can achieve up-to 50 percent reduction in wall clock time in some continuous control environments.
  arXiv  Detail & Related papers  (2022-06-02T17:27:49Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Model-based Reinforcement Learning for Continuous Control with Posterior Sampling [10.91557009257615]
  We study model-based posterior sampling for reinforcement learning (PSRL) in continuous state-action spaces. We present MPC-PSRL, a model-based posterior sampling algorithm with model predictive control for action selection.
  arXiv  Detail & Related papers  (2020-11-20T21:00:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.