Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs

• URL: http://arxiv.org/abs/2506.05387v2
• Date: Wed, 11 Jun 2025 16:08:29 GMT
• Title: Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs
• Authors: Jaydip Sen, Saptarshi Sengupta, Subhasis Dasgupta,
• Abstract summary: Adaptive Semantic-Aware Typicality Sampling (ASTS) is proposed as an improved version of the Locally Typical Sampling (LTS) algorithm.<n>ASTS ensures contextually coherent and diverse text generation while maintaining computational efficiency.<n> Experimental results demonstrate that ASTS outperforms existing sampling techniques by reducing repetition, enhancing semantic alignment, and improving fluency.
• Score: 0.9217021281095907
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This chapter explores advancements in decoding strategies for large language models (LLMs), focusing on enhancing the Locally Typical Sampling (LTS) algorithm. Traditional decoding methods, such as top-k and nucleus sampling, often struggle to balance fluency, diversity, and coherence in text generation. To address these challenges, Adaptive Semantic-Aware Typicality Sampling (ASTS) is proposed as an improved version of LTS, incorporating dynamic entropy thresholding, multi-objective scoring, and reward-penalty adjustments. ASTS ensures contextually coherent and diverse text generation while maintaining computational efficiency. Its performance is evaluated across multiple benchmarks, including story generation and abstractive summarization, using metrics such as perplexity, MAUVE, and diversity scores. Experimental results demonstrate that ASTS outperforms existing sampling techniques by reducing repetition, enhancing semantic alignment, and improving fluency.

Related papers

• Semantic uncertainty in advanced decoding methods for LLM generation [35.31962554915952]
  This study investigates semantic uncertainty in large language model (LLM) outputs across different decoding methods.<n>We analyze how different decoding strategies affect both the diversity and reliability of model outputs.
  arXiv  Detail & Related papers  (2025-06-17T10:09:29Z)
• Scaling Textual Gradients via Sampling-Based Momentum [59.94928977345951]
  The Textual Gradient Descent (TGD) framework has emerged as a promising data-driven approach.<n> scaling the number of training examples improves results but later degrades TGD's performance.<n>We propose Textual Gradient Descent with Momentum (TSGD-M) - a method that facilitates scalable-context learning by reweighting prompt sampling.
  arXiv  Detail & Related papers  (2025-05-31T05:35:45Z)
• Context-Enhanced Contrastive Search for Improved LLM Text Generation [1.7720658326850143]
  The paper proposes a novel enhancement of the well-known Contrastive Search algorithm, Context-Enhanced Contrastive Search ( CECS) with contextual calibration.<n>The proposed scheme introduces several novelties including dynamic contextual importance weighting, multi-level Contrastive Search, and adaptive temperature control, to optimize the balance between fluency, creativity, and precision.<n> Experimental results demonstrate significant improvements in both coherence and relevance of the generated texts by CECS outperforming the existing Contrastive Search techniques.
  arXiv  Detail & Related papers  (2025-04-22T13:00:14Z)
• Diversified Sampling Improves Scaling LLM inference [31.18762591875725]
  DivSampling is a novel and versatile sampling technique designed to enhance the diversity of candidate solutions.<n>Our theoretical analysis demonstrates that, under mild assumptions, the error rates of responses generated from diverse prompts are significantly lower compared to those produced by stationary prompts.
  arXiv  Detail & Related papers  (2025-02-16T07:37:58Z)
• Transformer-Driven Active Transfer Learning for Cross-Hyperspectral Image Classification [3.087068801861429]
  Hyperspectral image (HSI) classification presents inherent challenges due to high spectral dimensionality, significant domain shifts, and limited availability of labeled data.<n>We propose a novel Active Transfer Learning (ATL) framework built upon a Spatial-Spectral Transformer (SST) backbone.<n>The framework integrates multistage transfer learning with an uncertainty-diversity-driven active learning mechanism.
  arXiv  Detail & Related papers  (2024-11-27T07:53:39Z)
• Words Matter: Leveraging Individual Text Embeddings for Code Generation in CLIP Test-Time Adaptation [21.20806568508201]
  We show how to leverage class text information to mitigate distribution drifts encountered by vision-language models (VLMs) during test-time inference.<n>We propose to generate pseudo-labels for the test-time samples by exploiting generic class text embeddings as fixed centroids of a label assignment problem.<n>Experiments on multiple popular test-time adaptation benchmarks presenting diverse complexity empirically show the superiority of CLIP-OT.
  arXiv  Detail & Related papers  (2024-11-26T00:15:37Z)
• Balancing Diversity and Risk in LLM Sampling: How to Select Your Method and Parameter for Open-Ended Text Generation [60.493180081319785]
  We propose a systematic way to estimate the capacity of a truncation sampling method by considering the trade-off between diversity and risk at each decoding step.<n>Our work offers a comprehensive comparison of existing truncation sampling methods and serves as a practical user guideline for their parameter selection.
  arXiv  Detail & Related papers  (2024-08-24T14:14:32Z)
• Language Model Decoding as Direct Metrics Optimization [87.68281625776282]
  Current decoding methods struggle to generate texts that align with human texts across different aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts. We prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts.
  arXiv  Detail & Related papers  (2023-10-02T09:35:27Z)
• MacLaSa: Multi-Aspect Controllable Text Generation via Efficient Sampling from Compact Latent Space [110.85888003111653]
  Multi-aspect controllable text generation aims to generate fluent sentences that possess multiple desired attributes simultaneously. We introduce a novel approach for multi-aspect control, namely MacLaSa, that estimates compact latent space for multiple aspects. We show that MacLaSa outperforms several strong baselines on attribute relevance and textual quality while maintaining a high inference speed.
  arXiv  Detail & Related papers  (2023-05-22T07:30:35Z)
• SDA: Improving Text Generation with Self Data Augmentation [88.24594090105899]
  We propose to improve the standard maximum likelihood estimation (MLE) paradigm by incorporating a self-imitation-learning phase for automatic data augmentation. Unlike most existing sentence-level augmentation strategies, our method is more general and could be easily adapted to any MLE-based training procedure.
  arXiv  Detail & Related papers  (2021-01-02T01:15:57Z)
• Self-Adversarial Learning with Comparative Discrimination for Text Generation [111.18614166615968]
  We propose a novel self-adversarial learning (SAL) paradigm for improving GANs' performance in text generation. During training, SAL rewards the generator when its currently generated sentence is found to be better than its previously generated samples. Experiments on text generation benchmark datasets show that our proposed approach substantially improves both the quality and the diversity.
  arXiv  Detail & Related papers  (2020-01-31T07:50:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.