Entropy-informed Decoding: Adaptive Information-Driven Branching

• URL: http://arxiv.org/abs/2605.09745v1
• Date: Sun, 10 May 2026 20:44:47 GMT
• Title: Entropy-informed Decoding: Adaptive Information-Driven Branching
• Abstract summary: Large language models (LLMs) achieve remarkable generative performance, yet their output quality is dependent on the decoding strategy.<n>We introduce Entropy-informed decoding (EDEN), a model-agnostic decoding framework that adaptively allocates based on the model's own uncertainty.<n>EDEN consistently improves output quality over existing decoding strategies, achieving better accuracy-expansion trade-offs than fixed-width beam search.
• Score: 5.954754354938324
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Large language models (LLMs) achieve remarkable generative performance, yet their output quality is dependent on the decoding strategy. While sampling-based methods (e.g., top-k, nucleus) and search-and-select based methods (e.g., beam search, best-of-n, majority voting) can improve upon greedy decoding, both approaches suffer from limitations: sampling generally commits to a single path, while search often expends excessive computation regardless of task complexity. To address these, we introduce Entropy-informed decoding (EDEN), a plug-and-play, model-agnostic decoding framework that adaptively allocates computation based on the model's own uncertainty, approximating higher-width beam search with fewer expansions. At each generation step, EDEN estimates the entropy of the output token distribution and adjusts the branching factor monotonically with the entropy, expanding more candidates in high-entropy regions and following a greedier path in low-entropy regions, improving token efficiency. Experiments across complex tasks, including mathematical reasoning, code generation, and scientific questions, demonstrate that EDEN consistently improves output quality over existing decoding strategies, achieving better accuracy-expansion trade-offs than fixed-width beam search. By treating next-token selection as a noisy maximisation problem, we prove that branching factors monotone in entropy are guaranteed to find better (i.e. more probable) continuations than any fixed branching factor within the same total expansion budget, and derive explicit regret rates characterising the benefit of the adaptive allocation.

Related papers

• Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning [32.332197731504046]
  Decoding strategies play a central role in shaping the reasoning ability of large language models.<n>Traditional methods such as greedy decoding and beam search often suffer from error propagation.<n>We propose an entropy-guided decoding framework that introduces token-level adaptivity into generation.
  arXiv  Detail & Related papers  (2026-03-10T23:08:26Z)
• Optimizing Decoding Paths in Masked Diffusion Models by Quantifying Uncertainty [16.454646094266703]
  Masked Diffusion Models (MDMs) offer flexible, non-autoregressive generation, but this freedom introduces a challenge.<n>We are the first to formalize this issue, attributing variability in output quality to the cumulative predictive uncertainty along a generative path.<n>Our work establishes Denoising Entropy as a principled tool for understanding and controlling generation, effectively turning the uncertainty in MDMs from a liability into a key advantage for discovering high-quality solutions.
  arXiv  Detail & Related papers  (2025-12-24T18:59:51Z)
• Partial Information Decomposition via Normalizing Flows in Latent Gaussian Distributions [24.013410583573318]
  We propose a new gradient-based algorithm that substantially improves the computational efficiency of Gaussian PID (GPID)<n>We show that our proposed method provides more accurate and efficient PID estimates than existing baselines.<n>We evaluate a series of large-scale multimodal benchmarks to show its utility in real-world applications of quantifying PID in multimodal datasets.
  arXiv  Detail & Related papers  (2025-10-06T01:08:34Z)
• Unlocking Symbol-Level Precoding Efficiency Through Tensor Equivariant Neural Network [84.22115118596741]
  We propose an end-to-end deep learning (DL) framework with low inference complexity for symbol-level precoding.<n>We show that the proposed framework captures substantial performance gains of optimal SLP, while achieving an approximately 80-times speedup over conventional methods.
  arXiv  Detail & Related papers  (2025-10-02T15:15:50Z)
• LSR-MCTS: Alleviating Long Range Dependency in Code Generation [42.10272627826627]
  Large language models (LLMs) have significantly promoted the development of code generation task.<n>We propose the textbfLSR-MCTS algorithm, which leverages MCTS to determine the code line-by-line and select the optimal path.
  arXiv  Detail & Related papers  (2025-04-10T04:03:25Z)
• Dynamic Search for Inference-Time Alignment in Diffusion Models [87.35944312589424]
  We frame inference-time alignment in diffusion as a search problem and propose Dynamic Search for Diffusion (DSearch)<n>DSearch subsamples from denoising processes and approximates intermediate node rewards.<n>It also dynamically adjusts beam width and tree expansion to efficiently explore high-reward generations.
  arXiv  Detail & Related papers  (2025-03-03T20:32:05Z)
• Amortizing intractable inference in large language models [56.92471123778389]
  We use amortized Bayesian inference to sample from intractable posterior distributions. We empirically demonstrate that this distribution-matching paradigm of LLM fine-tuning can serve as an effective alternative to maximum-likelihood training. As an important application, we interpret chain-of-thought reasoning as a latent variable modeling problem.
  arXiv  Detail & Related papers  (2023-10-06T16:36:08Z)
• Conditional Denoising Diffusion for Sequential Recommendation [62.127862728308045]
  Two prominent generative models, Generative Adversarial Networks (GANs) and Variational AutoEncoders (VAEs) GANs suffer from unstable optimization, while VAEs are prone to posterior collapse and over-smoothed generations. We present a conditional denoising diffusion model, which includes a sequence encoder, a cross-attentive denoising decoder, and a step-wise diffuser.
  arXiv  Detail & Related papers  (2023-04-22T15:32:59Z)
• Local policy search with Bayesian optimization [73.0364959221845]
  Reinforcement learning aims to find an optimal policy by interaction with an environment. Policy gradients for local search are often obtained from random perturbations. We develop an algorithm utilizing a probabilistic model of the objective function and its gradient.
  arXiv  Detail & Related papers  (2021-06-22T16:07:02Z)
• Adaptive Sampling for Best Policy Identification in Markov Decision Processes [79.4957965474334]
  We investigate the problem of best-policy identification in discounted Markov Decision (MDPs) when the learner has access to a generative model. The advantages of state-of-the-art algorithms are discussed and illustrated.
  arXiv  Detail & Related papers  (2020-09-28T15:22:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.