Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation

• URL: http://arxiv.org/abs/2410.06097v1
• Date: Tue, 8 Oct 2024 14:51:03 GMT
• Title: Decoding Decoded: Understanding Hyperparameter Effects in Open-Ended Text Generation
• Authors: Esteban Garces Arias, Meimingwei Li, Christian Heumann, Matthias Aßenmacher,
• Abstract summary: Decoding strategies for large language models (LLMs) are a critical but often underexplored aspect of text generation tasks. We present a large-scale, comprehensive analysis of how hyper parameter selection affects text quality in open-ended text generation.
• Score: 0.22499166814992438
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decoding strategies for large language models (LLMs) are a critical but often underexplored aspect of text generation tasks. Since LLMs produce probability distributions over the entire vocabulary, various decoding methods have been developed to transform these probabilities into coherent and fluent text, each with its own set of hyperparameters. In this study, we present a large-scale, comprehensive analysis of how hyperparameter selection affects text quality in open-ended text generation across multiple LLMs, datasets, and evaluation metrics. Through an extensive sensitivity analysis, we provide practical guidelines for hyperparameter tuning and demonstrate the substantial influence of these choices on text quality. Using three established datasets, spanning factual domains (e.g., news) and creative domains (e.g., fiction), we show that hyperparameter tuning significantly impacts generation quality, though its effects vary across models and tasks. We offer in-depth insights into these effects, supported by both human evaluations and a synthesis of widely-used automatic evaluation metrics.

Related papers

• Towards Better Open-Ended Text Generation: A Multicriteria Evaluation Framework [0.1979158763744267]
  Open-ended text generation has become a prominent task in natural language processing. Decoding methods often excel in some metrics while underperforming in others. We present novel ranking strategies within this multicriteria framework.
  arXiv  Detail & Related papers  (2024-10-24T11:32:01Z)
• Detecting Machine-Generated Long-Form Content with Latent-Space Variables [54.07946647012579]
  Existing zero-shot detectors primarily focus on token-level distributions, which are vulnerable to real-world domain shifts. We propose a more robust method that incorporates abstract elements, such as event transitions, as key deciding factors to detect machine versus human texts.
  arXiv  Detail & Related papers  (2024-10-04T18:42:09Z)
• Adaptive Contrastive Search: Uncertainty-Guided Decoding for Open-Ended Text Generation [0.20971479389679337]
  We introduce adaptive contrastive search, a novel decoding strategy extending contrastive search. Our findings indicate performance enhancement in both aspects, across different model architectures and datasets.
  arXiv  Detail & Related papers  (2024-07-26T12:23:54Z)
• SMLT-MUGC: Small, Medium, and Large Texts -- Machine versus User-Generated Content Detection and Comparison [2.7147912878168303]
  We compare the performance of machine learning algorithms on four datasets: (1) small (tweets from Election, FIFA, and Game of Thrones), (2) medium (Wikipedia introductions and PubMed abstracts), and (3) large (OpenAI web text dataset) Our results indicate that LLMs with very large parameters (such as the XL-1542 variant of GPT2 with 1542 million parameters) were harder to detect using traditional machine learning methods. We examine the characteristics of human and machine-generated texts across multiple dimensions, including linguistics, personality, sentiment, bias, and morality.
  arXiv  Detail & Related papers  (2024-06-28T22:19:01Z)
• LLM can Achieve Self-Regulation via Hyperparameter Aware Generation [88.69052513433603]
  Large Language Models (LLMs) employ diverse decoding strategies to control the generated text. Are LLMs conscious of the existence of these decoding strategies and capable of regulating themselves? We propose a novel text generation paradigm termed Hyperparameter Aware Generation (HAG)
  arXiv  Detail & Related papers  (2024-02-17T11:18:22Z)
• ToBlend: Token-Level Blending With an Ensemble of LLMs to Attack AI-Generated Text Detection [6.27025292177391]
  ToBlend is a novel token-level ensemble text generation method to challenge the robustness of current AI-content detection approaches. We find ToBlend significantly drops the performance of most mainstream AI-content detection methods.
  arXiv  Detail & Related papers  (2024-02-17T02:25:57Z)
• A Thorough Examination of Decoding Methods in the Era of LLMs [72.65956436513241]
  Decoding methods play an indispensable role in converting language models from next-token predictors into practical task solvers. This paper provides a comprehensive and multifaceted analysis of various decoding methods within the context of large language models. Our findings reveal that decoding method performance is notably task-dependent and influenced by factors such as alignment, model size, and quantization.
  arXiv  Detail & Related papers  (2024-02-10T11:14:53Z)
• Successor Features for Efficient Multisubject Controlled Text Generation [48.37713738712319]
  We introduce SF-GEN, which is grounded in two primary concepts: successor features (SFs) and language model rectification. SF-GEN seamlessly integrates the two to enable dynamic steering of text generation with no need to alter the LLM's parameters. To the best of our knowledge, our research represents the first application of successor features in text generation.
  arXiv  Detail & Related papers  (2023-11-03T00:17:08Z)
• Generative Judge for Evaluating Alignment [84.09815387884753]
  We propose a generative judge with 13B parameters, Auto-J, designed to address these challenges. Our model is trained on user queries and LLM-generated responses under massive real-world scenarios. Experimentally, Auto-J outperforms a series of strong competitors, including both open-source and closed-source models.
  arXiv  Detail & Related papers  (2023-10-09T07:27:15Z)
• Language Model Decoding as Direct Metrics Optimization [87.68281625776282]
  Current decoding methods struggle to generate texts that align with human texts across different aspects. In this work, we frame decoding from a language model as an optimization problem with the goal of strictly matching the expected performance with human texts. We prove that this induced distribution is guaranteed to improve the perplexity on human texts, which suggests a better approximation to the underlying distribution of human texts.
  arXiv  Detail & Related papers  (2023-10-02T09:35:27Z)
• Reverse-Engineering Decoding Strategies Given Blackbox Access to a Language Generation System [73.52878118434147]
  We present methods to reverse-engineer the decoding method used to generate text. Our ability to discover which decoding strategy was used has implications for detecting generated text.
  arXiv  Detail & Related papers  (2023-09-09T18:19:47Z)
• On the Possibilities of AI-Generated Text Detection [76.55825911221434]
  We argue that as machine-generated text approximates human-like quality, the sample size needed for detection bounds increases. We test various state-of-the-art text generators, including GPT-2, GPT-3.5-Turbo, Llama, Llama-2-13B-Chat-HF, and Llama-2-70B-Chat-HF, against detectors, including oBERTa-Large/Base-Detector, GPTZero.
  arXiv  Detail & Related papers  (2023-04-10T17:47:39Z)
• An Analysis of the Effects of Decoding Algorithms on Fairness in Open-Ended Language Generation [77.44921096644698]
  We present a systematic analysis of the impact of decoding algorithms on LM fairness. We analyze the trade-off between fairness, diversity and quality.
  arXiv  Detail & Related papers  (2022-10-07T21:33:34Z)
• On Decoding Strategies for Neural Text Generators [73.48162198041884]
  We study the interaction between language generation tasks and decoding strategies. We measure changes in attributes of generated text as a function of both decoding strategy and task. Our results reveal both previously-observed and surprising findings.
  arXiv  Detail & Related papers  (2022-03-29T16:25:30Z)
• GenAug: Data Augmentation for Finetuning Text Generators [21.96895115572357]
  We propose and evaluate various augmentation methods, including some that incorporate external knowledge, for finetuning GPT-2 on a subset of Yelp Reviews. Our experiments demonstrate that insertion of character-level synthetic noise and keyword replacement with hypernyms are effective augmentation methods.
  arXiv  Detail & Related papers  (2020-10-05T05:46:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.