Exploring the Impact of Temperature on Large Language Models:Hot or Cold?

• URL: http://arxiv.org/abs/2506.07295v1
• Date: Sun, 08 Jun 2025 21:36:26 GMT
• Title: Exploring the Impact of Temperature on Large Language Models:Hot or Cold?
• Authors: Lujun Li, Lama Sleem, Niccolo' Gentile, Geoffrey Nichil, Radu State,
• Abstract summary: We evaluate the impact of temperature in the range of 0 to 2 on data sets designed to assess six different capabilities.<n>Our findings reveal skill-specific effects of temperature on model performance, highlighting the complexity of optimal temperature selection.<n>We propose a BERT-based temperature selector that takes advantage of these observed effects to identify the optimal temperature for a given prompt.
• Score: 9.70280446429164
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: The sampling temperature, a critical hyperparameter in large language models (LLMs), modifies the logits before the softmax layer, thereby reshaping the distribution of output tokens. Recent studies have challenged the Stochastic Parrots analogy by demonstrating that LLMs are capable of understanding semantics rather than merely memorizing data and that randomness, modulated by sampling temperature, plays a crucial role in model inference. In this study, we systematically evaluated the impact of temperature in the range of 0 to 2 on data sets designed to assess six different capabilities, conducting statistical analyses on open source models of three different sizes: small (1B--4B), medium (6B--13B), and large (40B--80B). Our findings reveal distinct skill-specific effects of temperature on model performance, highlighting the complexity of optimal temperature selection in practical applications. To address this challenge, we propose a BERT-based temperature selector that takes advantage of these observed effects to identify the optimal temperature for a given prompt. We demonstrate that this approach can significantly improve the performance of small and medium models in the SuperGLUE datasets. Furthermore, our study extends to FP16 precision inference, revealing that temperature effects are consistent with those observed in 4-bit quantized models. By evaluating temperature effects up to 4.0 in three quantized models, we find that the Mutation Temperature -- the point at which significant performance changes occur -- increases with model size.

Related papers

• Exploring the Impact of Temperature Scaling in Softmax for Classification and Adversarial Robustness [8.934328206473456]
  This study delves into the often-overlooked parameter within the softmax function, known as "temperature"<n>Our empirical studies, adopting convolutional neural networks and transformers, reveal that moderate temperatures generally introduce better overall performance.<n>For the first time, we discover a surprising benefit of elevated temperatures: enhanced model robustness against common corruption, natural perturbation, and non-targeted adversarial attacks like Projected Gradient Descent.
  arXiv  Detail & Related papers  (2025-02-28T00:07:45Z)
• Optimizing Temperature for Language Models with Multi-Sample Inference [47.14991144052361]
  This paper addresses the challenge of automatically identifying the (near)-optimal temperature for different large language models.<n>We provide a comprehensive analysis of temperature's role in performance optimization, considering variations in model architectures, datasets, task types, model sizes, and predictive accuracy.<n>We propose a novel entropy-based metric for automated temperature optimization, which consistently outperforms fixed-temperature baselines.
  arXiv  Detail & Related papers  (2025-02-07T19:35:25Z)
• Adaptive Decoding via Latent Preference Optimization [55.70602730588745]
  We introduce Adaptive Decoding, a layer added to the model to select the sampling temperature dynamically at inference time. Our method outperforms all fixed decoding temperatures across a range of tasks that require different temperatures.
  arXiv  Detail & Related papers  (2024-11-14T18:31:39Z)
• Calibrating Language Models with Adaptive Temperature Scaling [58.056023173579625]
  We introduce Adaptive Temperature Scaling (ATS), a post-hoc calibration method that predicts a temperature scaling parameter for each token prediction. ATS improves calibration by over 10-50% across three downstream natural language evaluation benchmarks compared to prior calibration methods.
  arXiv  Detail & Related papers  (2024-09-29T22:54:31Z)
• Deep generative modelling of canonical ensemble with differentiable thermal properties [0.9421843976231371]
  We propose a variational modelling method with differentiable temperature for canonical ensembles. Using a deep generative model, the free energy is estimated and minimized simultaneously in a continuous temperature range. The training process requires no dataset, and works with arbitrary explicit density generative models.
  arXiv  Detail & Related papers  (2024-04-29T03:41:49Z)
• A Three-regime Model of Network Pruning [47.92525418773768]
  We use temperature-like and load-like parameters to model the impact of neural network (NN) training hyper parameters on pruning performance. A key empirical result we identify is a sharp transition phenomenon: depending on the value of a load-like parameter in the pruned model, increasing the value of a temperature-like parameter in the pre-pruned model may either enhance or impair subsequent pruning performance. Our model reveals that the dichotomous effect of high temperature is associated with transitions between distinct types of global structures in the post-pruned model.
  arXiv  Detail & Related papers  (2023-05-28T08:09:25Z)
• Long Horizon Temperature Scaling [90.03310732189543]
  Long Horizon Temperature Scaling (LHTS) is a novel approach for sampling from temperature-scaled joint distributions. We derive a temperature-dependent LHTS objective, and show that finetuning a model on a range of temperatures produces a single model capable of generation with a controllable long horizon temperature parameter.
  arXiv  Detail & Related papers  (2023-02-07T18:59:32Z)
• Fine-tune your Classifier: Finding Correlations With Temperature [2.071516130824992]
  We analyze the impact of temperature on classification tasks by describing a dataset as a set of statistics computed on representations. We study the correlation between these extracted statistics and the observed optimal temperatures.
  arXiv  Detail & Related papers  (2022-10-18T09:48:46Z)
• Uhlmann Fidelity and Fidelity Susceptibility for Integrable Spin Chains at Finite Temperature: Exact Results [68.8204255655161]
  We show that the proper inclusion of the odd parity subspace leads to the enhancement of maximal fidelity susceptibility in the intermediate range of temperatures. The correct low-temperature behavior is captured by an approximation involving the two lowest many-body energy eigenstates.
  arXiv  Detail & Related papers  (2021-05-11T14:08:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.