Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding

• URL: http://arxiv.org/abs/2606.22511v1
• Date: Sun, 21 Jun 2026 14:04:51 GMT
• Title: Breaking the Likelihood Trap: Variance-Calibrated Modulation for Large Language Model Decoding
• Abstract summary: Variance-Calibrated Modulation (VCM) is a training-free pre-decoding intervention that reshapes the probability distribution before truncation.<n>Across open-ended generation, factual QA, and mathematical reasoning, VCM consistently mitigates the likelihood trap.
• Score: 4.534022278426311
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In open-ended generation, LLMs frequently fall into the "likelihood trap", marked by repetitive degeneration and vocabulary dullness, creating a discrepancy between machine-generated and human-written text. While post-hoc tail truncation (e.g., Top-$p$, Min-$p$) avoids sampling from the unreliable tail, it can over-sample from the uncalibrated head and misalign generation with human lexical preferences; fixed scalar repetition penalties likewise ignore variation in logit scale across inference steps, potentially disrupting semantic coherence. To address both limitations, we propose Variance-Calibrated Modulation (VCM), a training-free pre-decoding intervention that reshapes the probability distribution before truncation through two dynamic mechanisms: (1) Contextual Searchlight via PMI, which suppresses global stopwords while elevating context-evoked tokens, and (2) Adaptive Self-Debiasing, which uses real-time logit standard deviation for scale-invariant penalization. Across open-ended generation, factual QA, and mathematical reasoning, VCM consistently mitigates the likelihood trap. With negligible computational overhead, VCM integrates with existing decoding strategies, improving diversity, coherence, and, particularly at higher decoding temperatures, reasoning accuracy.

Related papers

• Beyond Homoscedasticity: Decoupled Uncertainty Optimization for Deep Imbalanced Regression [7.011984503405774]
  Deep Imbalanced Regression is pervasive in continuous prediction tasks across diverse modalities.<n>We propose DUO, an uncertainty-aware long-tailed regression framework.<n>We show that DUO achieves the best few-shot bMAE and GM on IMDB-WIKI-DIR, AgeDB-DIR, and AAV2-DIR while remaining competitive on few-shot MAE.
  arXiv  Detail & Related papers  (2026-09-04T11:06:57Z)
• Escaping Confidence Trap: Evolutionary Decoding for Mathematical Reasoning in Diffusion LLMs [69.52853771994451]
  We analyze the decoding trajectories of LLaDA 2.0 and identify a recurring diffusion confidence trap.<n>We propose Evolutionary Decoding, a training-free test-time scaling framework that views diffusion decoding as an evolutionary process.
  arXiv  Detail & Related papers  (2026-08-01T11:48:25Z)
• Prototype-Anchored Generalized Manifold Regression for Unknown-Domain Object Detection [58.25418970608328]
  We study Single-Domain Generalized Object Detection (Single-DGOD), which aims to transfer a detector trained on a single source domain to multiple unseen domains.<n>Existing methods mainly rely on simulation-driven strategies, such as data augmentation or textual prompts, to enlarge the training distribution.<n>We propose Manifold Regression with Visual-Text Dual Chain-of-Thought (MR-DCoT), which formulates unknown-domain generalization as a manifold regression problem.
  arXiv  Detail & Related papers  (2026-07-08T09:27:15Z)
• C$^{2}$R: Cross-sample Consistency Regularization Mitigates Feature Splitting and Absorption in Sparse Autoencoders [56.67585330425431]
  Sparse Autoencoders (SAEs) are widely used to interpret large language models by decomposing activations into sparse, human-understandable features.<n>These issues stem from inconsistent latent assignment across samples.<n>We introduce C$2$R, which penalizes the co-activation of directionally similar latents.
  arXiv  Detail & Related papers  (2026-06-29T17:45:31Z)
• Reconstruction of detector error model for quantum error correction [6.505803823301456]
  We introduce a globally consistent framework to invert experimental syndrome statistics directly into discrete physical hypergraphs.<n>We show that exact continuous parameter extraction in dense codes is limited by a textitvariance cascade, where absolute statistical variance accumulates linearly from high to low-degree mechanisms.<n>This provides a practical approach for characterizing and decoding highly correlated noise in realistic quantum hardware.
  arXiv  Detail & Related papers  (2026-06-15T06:51:48Z)
• Relaxation-Informed Training of Neural Network Surrogate Models [3.9722979176564763]
  ReLU neural networks trained surrogate models can be embedded exactly in mixed-integer linear programs (MILPs)<n>The tractability of the resulting MILP depends on structural properties of the network.<n>We show that the proposed regularizers can reduce MILP times by up to four orders of magnitude to an unregularized baseline.
  arXiv  Detail & Related papers  (2026-04-24T17:46:55Z)
• Think Twice Before You Write -- an Entropy-based Decoding Strategy to Enhance LLM Reasoning [32.332197731504046]
  Decoding strategies play a central role in shaping the reasoning ability of large language models.<n>Traditional methods such as greedy decoding and beam search often suffer from error propagation.<n>We propose an entropy-guided decoding framework that introduces token-level adaptivity into generation.
  arXiv  Detail & Related papers  (2026-03-10T23:08:26Z)
• Fast Controlled Generation from Language Models with Adaptive Weighted Rejection Sampling [90.86991492288487]
  evaluating constraint on every token can be prohibitively expensive.<n> LCD can distort the global distribution over strings, sampling tokens based only on local information.<n>We show that our approach is superior to state-of-the-art baselines.
  arXiv  Detail & Related papers  (2025-04-07T18:30:18Z)
• Autoregressive Speech Synthesis without Vector Quantization [135.4776759536272]
  We present MELLE, a novel continuous-valued token based language modeling approach for text-to-speech synthesis (TTS)<n>MELLE autoregressively generates continuous mel-spectrogram frames directly from text condition.<n>MELLE mitigates robustness issues by avoiding the inherent flaws of sampling vector-quantized codes.
  arXiv  Detail & Related papers  (2024-07-11T14:36:53Z)
• Outlier-robust neural network training: variation regularization meets trimmed loss to prevent functional breakdown [2.5628953713168685]
  We tackle the challenge of outlier-robust predictive modeling using highly expressive neural networks.<n>Our approach integrates two key components: (1) a transformed trimmed loss (TTL), and (2) higher-order variation regularization (HOVR), which imposes smoothness constraints on the prediction function.
  arXiv  Detail & Related papers  (2023-08-04T12:57:13Z)
• Tailoring Language Generation Models under Total Variation Distance [55.89964205594829]
  The standard paradigm of neural language generation adopts maximum likelihood estimation (MLE) as the optimizing method. We develop practical bounds to apply it to language generation. We introduce the TaiLr objective that balances the tradeoff of estimating TVD.
  arXiv  Detail & Related papers  (2023-02-26T16:32:52Z)
• Tail-to-Tail Non-Autoregressive Sequence Prediction for Chinese Grammatical Error Correction [49.25830718574892]
  We present a new framework named Tail-to-Tail (textbfTtT) non-autoregressive sequence prediction. Considering that most tokens are correct and can be conveyed directly from source to target, and the error positions can be estimated and corrected. Experimental results on standard datasets, especially on the variable-length datasets, demonstrate the effectiveness of TtT in terms of sentence-level Accuracy, Precision, Recall, and F1-Measure.
  arXiv  Detail & Related papers  (2021-06-03T05:56:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.