Context-Enhanced Contrastive Search for Improved LLM Text Generation

• URL: http://arxiv.org/abs/2504.21020v1
• Date: Tue, 22 Apr 2025 13:00:14 GMT
• Title: Context-Enhanced Contrastive Search for Improved LLM Text Generation
• Authors: Jaydip Sen, Rohit Pandey, Hetvi Waghela,
• Abstract summary: The paper proposes a novel enhancement of the well-known Contrastive Search algorithm, Context-Enhanced Contrastive Search ( CECS) with contextual calibration.<n>The proposed scheme introduces several novelties including dynamic contextual importance weighting, multi-level Contrastive Search, and adaptive temperature control, to optimize the balance between fluency, creativity, and precision.<n> Experimental results demonstrate significant improvements in both coherence and relevance of the generated texts by CECS outperforming the existing Contrastive Search techniques.
• Score: 1.7720658326850143
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recently, Large Language Models (LLMs) have demonstrated remarkable advancements in Natural Language Processing (NLP). However, generating high-quality text that balances coherence, diversity, and relevance remains challenging. Traditional decoding methods, such as bean search and top-k sampling, often struggle with either repetitive or incoherent outputs, particularly in tasks that require long-form text generation. To address these limitations, the paper proposes a novel enhancement of the well-known Contrastive Search algorithm, Context-Enhanced Contrastive Search (CECS) with contextual calibration. The proposed scheme introduces several novelties including dynamic contextual importance weighting, multi-level Contrastive Search, and adaptive temperature control, to optimize the balance between fluency, creativity, and precision. The performance of CECS is evaluated using several standard metrics such as BLEU, ROUGE, and semantic similarity. Experimental results demonstrate significant improvements in both coherence and relevance of the generated texts by CECS outperforming the existing Contrastive Search techniques. The proposed algorithm has several potential applications in the real world including legal document drafting, customer service chatbots, and content marketing.

Related papers

• Advancing Decoding Strategies: Enhancements in Locally Typical Sampling for LLMs [0.9217021281095907]
  Adaptive Semantic-Aware Typicality Sampling (ASTS) is proposed as an improved version of the Locally Typical Sampling (LTS) algorithm.<n>ASTS ensures contextually coherent and diverse text generation while maintaining computational efficiency.<n> Experimental results demonstrate that ASTS outperforms existing sampling techniques by reducing repetition, enhancing semantic alignment, and improving fluency.
  arXiv  Detail & Related papers  (2025-06-03T14:25:23Z)
• Optimizing Multi-Stage Language Models for Effective Text Retrieval [0.0]
  We introduce a novel two-phase text retrieval pipeline optimized for Japanese legal datasets.<n>Our method leverages advanced language models to achieve state-of-the-art performance.<n>To further enhance robustness and adaptability, we incorporate an ensemble model that integrates multiple retrieval strategies.
  arXiv  Detail & Related papers  (2024-12-26T16:05:19Z)
• Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework [0.1979158763744267]
  Open-ended text generation has become a prominent task in natural language processing.<n> evaluating the quality of these models and the employed decoding strategies remains challenging.<n>This paper proposes novel methods for both relative and absolute rankings of decoding methods.
  arXiv  Detail & Related papers  (2024-10-24T11:32:01Z)
• Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation [0.20971479389679337]
  We introduce adaptive contrastive search, a novel decoding strategy extending contrastive search. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets.
  arXiv  Detail & Related papers  (2024-07-26T12:23:54Z)
• Fidelity-Enriched Contrastive Search: Reconciling the Faithfulness-Diversity Trade-Off in Text Generation [21.096737598952853]
  We propose a new decoding method called Fidelity-Enriched Contrastive Search (FECS) FECS promotes tokens that are semantically similar to the provided source while penalizing repetitiveness in the generated text. Results show that FECS consistently enhances faithfulness across various language model sizes while maintaining output diversity comparable to well-performing decoding algorithms.
  arXiv  Detail & Related papers  (2023-10-23T14:27:45Z)
• Language Model Decoding as Direct Metrics Optimization [87.68281625776282]
  Current decoding methods struggle to generate texts that align with human texts across different aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts. We prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts.
  arXiv  Detail & Related papers  (2023-10-02T09:35:27Z)
• ESTextSpotter: Towards Better Scene Text Spotting with Explicit Synergy in Transformer [88.61312640540902]
  We introduce Explicit Synergy-based Text Spotting Transformer framework (ESTextSpotter) Our model achieves explicit synergy by modeling discriminative and interactive features for text detection and recognition within a single decoder. Experimental results demonstrate that our model significantly outperforms previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-08-20T03:22:23Z)
• An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation [77.44921096644698]
  We present a systematic analysis of the impact of decoding algorithms on LM fairness. We analyze the trade-off between fairness, diversity and quality.
  arXiv  Detail & Related papers  (2022-10-07T21:33:34Z)
• Composable Text Controls in Latent Space with ODEs [97.12426987887021]
  This paper proposes a new efficient approach for composable text operations in the compact latent space of text. By connecting pretrained LMs to the latent space through efficient adaption, we then decode the sampled vectors into desired text sequences. Experiments show that composing those operators within our approach manages to generate or edit high-quality text.
  arXiv  Detail & Related papers  (2022-08-01T06:51:45Z)
• Faithfulness in Natural Language Generation: A Systematic Survey of Analysis, Evaluation and Optimization Methods [48.47413103662829]
  Natural Language Generation (NLG) has made great progress in recent years due to the development of deep learning techniques such as pre-trained language models. However, the faithfulness problem that the generated text usually contains unfaithful or non-factual information has become the biggest challenge.
  arXiv  Detail & Related papers  (2022-03-10T08:28:32Z)
• Obtaining Better Static Word Embeddings Using Contextual Embedding Models [53.86080627007695]
  Our proposed distillation method is a simple extension of CBOW-based training. As a side-effect, our approach also allows a fair comparison of both contextual and static embeddings.
  arXiv  Detail & Related papers  (2021-06-08T12:59:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.