Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes

• URL: http://arxiv.org/abs/2603.14567v1
• Date: Sun, 15 Mar 2026 19:39:51 GMT
• Title: Top-b: Entropic Regulation of Relative Probability Bands in Autoregressive Language Processes
• Abstract summary: Top-b is a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient.<n>We show that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy.
• Score: 16.569049230416123
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Probabilistic language generators are theoretically modeled as discrete stochastic processes, yet standard decoding strategies (Top-k, Top-p) impose static truncation rules that fail to accommodate the dynamic information density of natural language. This misalignment often forces a suboptimal trade-off: static bounds are either too restrictive for high-entropy creative generation or too permissive for low-entropy logical reasoning. In this work, we formalize the generation process as a trajectory through a relative probability manifold. We introduce Top-b (Adaptive Relative Band Sampling), a decoding strategy that regulates the candidate set via a dynamic bandwidth coefficient coupled strictly to the instantaneous Shannon entropy of the model's distribution. We provide a theoretical framework demonstrating that Top-b acts as a variance-minimizing operator on the tail distribution. Empirical validation on GPQA and GSM8K benchmarks indicates that Top-b significantly reduces generation entropy and inter-decoding variance while maintaining competitive reasoning accuracy, effectively approximating a self-regulating control system for autoregressive generation.

Related papers

• Mitigating Bias in Locally Constrained Decoding via Tractable Proposals [73.78736135699953]
  Existing locally constrained decoding approaches enforce constraints by myopically masking out next tokens.<n>Recent work uses sequential Monte Carlo methods to mitigate such biases, but designing effective proposal distributions or potential functions remains a key challenge.<n>We propose a generic approach to construct proposals and potentials for SMC sampling from $p_mathrmlm( cdot mid mathrmconstraint)$.
  arXiv  Detail & Related papers  (2026-06-01T08:58:18Z)
• Matching Features, Not Tokens: Energy-Based Fine-Tuning of Language Models [102.20309135516186]
  Cross-entropy (CE) training provides dense and scalable supervision for language models.<n>We introduce a feature-matching objective for language-model fine-tuning.<n>To optimize this objective efficiently, we propose energy-based fine-tuning.
  arXiv  Detail & Related papers  (2026-03-12T17:57:50Z)
• Entropy-Controlled Flow Matching [0.08460698440162889]
  We propose a constrained variational principle over continuity-equation paths enforcing a global entropy-rate budget d/dt H(mu_t) >= -lambda.<n>We obtain certificate-style mode-coverage and density-floor guarantees with Lipschitz, and construct near-optimal counterexamples for unconstrained flow matching.
  arXiv  Detail & Related papers  (2026-02-25T06:07:01Z)
• GTS: Inference-Time Scaling of Latent Reasoning with a Learnable Gaussian Thought Sampler [54.10960908347221]
  We model latent thought exploration as conditional sampling from learnable densities and instantiate this idea as a Gaussian Thought Sampler (GTS)<n>GTS predicts context-dependent perturbation distributions over continuous reasoning states and is trained with GRPO-style policy optimization while keeping the backbone frozen.
  arXiv  Detail & Related papers  (2026-02-15T09:57:47Z)
• Entropy-Reservoir Bregman Projection: An Information-Geometric Unification of Model Collapse [3.533187668612022]
  We present EntropyReser Bregman Projection- ERBP, an information-geometric framework that unifies these phenomena.<n>Our theory yields a necessary condition for collapse, (ii) a sufficient condition that guarantees a non-language entropy floor, and (iii) closed-form rates that depend on sample size.
  arXiv  Detail & Related papers  (2025-12-16T19:50:03Z)
• Worst-case generation via minimax optimization in Wasserstein space [19.645939141861543]
  Worst-case generation plays a critical role in evaluating robustness and stress-testing systems under distribution shifts.<n>We develop a generative modeling framework for worst-case generation for a pre-specified risk.
  arXiv  Detail & Related papers  (2025-12-09T02:11:08Z)
• Optimizing Chain-of-Thought Reasoners via Gradient Variance Minimization in Rejection Sampling and RL [20.177871969184004]
  Chain-of-thought (CoT) reasoning can be formalized as a latent variable problem, where the model needs to generate intermediate reasoning steps.<n>Prior approaches such as iterative reward-ranked fine-tuning fail to account for variability in difficulty and convergence behavior.<n>We propose GVMRAFT, a prompt-specific Dynamic Sample Allocation Strategy to minimize gradient variance under a computational budget constraint.
  arXiv  Detail & Related papers  (2025-05-05T06:26:00Z)
• Constrained Discrete Diffusion [61.81569616239755]
  This paper introduces Constrained Discrete Diffusion (CDD), a novel integration of differentiable constraint optimization within the diffusion process.<n>CDD directly imposes constraints into the discrete diffusion sampling process, resulting in a training-free and effective approach.
  arXiv  Detail & Related papers  (2025-03-12T19:48:12Z)
• Stochastic Control for Fine-tuning Diffusion Models: Optimality, Regularity, and Convergence [19.484676783876306]
  Diffusion models have emerged as powerful tools for generative modeling.<n>We propose a control framework for fine-tuning diffusion models.<n>We show that PI-FT achieves global convergence at a linear rate.
  arXiv  Detail & Related papers  (2024-12-24T04:55:46Z)
• A Pseudo-Semantic Loss for Autoregressive Models with Logical Constraints [87.08677547257733]
  Neuro-symbolic AI bridges the gap between purely symbolic and neural approaches to learning. We show how to maximize the likelihood of a symbolic constraint w.r.t the neural network's output distribution. We also evaluate our approach on Sudoku and shortest-path prediction cast as autoregressive generation.
  arXiv  Detail & Related papers  (2023-12-06T20:58:07Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.