The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence

• URL: http://arxiv.org/abs/2505.23420v1
• Date: Thu, 29 May 2025 13:10:57 GMT
• Title: The Warmup Dilemma: How Learning Rate Strategies Impact Speech-to-Text Model Convergence
• Authors: Marco Gaido, Sara Papi, Luisa Bentivogli, Alessio Brutti, Mauro Cettolo, Roberto Gretter, Marco Matassoni, Mohamed Nabih, Matteo Negri,
• Abstract summary: Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence.<n>OWSM designed a double linear warmup of the learning rate, increasing it to a very small value in the first phase before updating it to a higher value in the second phase.<n>This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.
• Score: 27.27404080214069
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Training large-scale models presents challenges not only in terms of resource requirements but also in terms of their convergence. For this reason, the learning rate (LR) is often decreased when the size of a model is increased. Such a simple solution is not enough in the case of speech-to-text (S2T) trainings, where evolved and more complex variants of the Transformer architecture -- e.g., Conformer or Branchformer -- are used in light of their better performance. As a workaround, OWSM designed a double linear warmup of the LR, increasing it to a very small value in the first phase before updating it to a higher value in the second phase. While this solution worked well in practice, it was not compared with alternative solutions, nor was the impact on the final performance of different LR warmup schedules studied. This paper fills this gap, revealing that i) large-scale S2T trainings demand a sub-exponential LR warmup, and ii) a higher LR in the warmup phase accelerates initial convergence, but it does not boost final performance.

Related papers

• SimpleGVR: A Simple Baseline for Latent-Cascaded Video Super-Resolution [55.14432034345353]
  We study key design principles for latter cascaded video super-resolution models, which are underexplored currently.<n>First, we propose two strategies to generate training pairs that better mimic the output characteristics of the base model, ensuring alignment between the VSR model and its upstream generator.<n>Second, we provide critical insights into VSR model behavior through systematic analysis of (1) timestep sampling strategies, (2) noise augmentation effects on low-resolution (LR) inputs.
  arXiv  Detail & Related papers  (2025-06-24T17:57:26Z)
• Walk Before You Run! Concise LLM Reasoning via Reinforcement Learning [10.255235456427037]
  We propose a simple yet effective two-stage reinforcement learning framework for achieving concise reasoning in Large Language Models (LLMs)<n>The first stage, using more training steps, aims to incentivize the model's reasoning capabilities via Group Relative Policy Optimization.<n>The second stage, using fewer training steps, explicitly enforces conciseness and improves efficiency via Length-aware Group Relative Policy Optimization.
  arXiv  Detail & Related papers  (2025-05-27T13:29:51Z)
• From Logistic Regression to the Perceptron Algorithm: Exploring Gradient Descent with Large Step Sizes [57.93371273485736]
  We focus on the classification problem with a separable dataset.<n>Recent studies have observed that LR+GD can find a solution with arbitrarily large step sizes.
  arXiv  Detail & Related papers  (2024-12-11T14:43:39Z)
• Less is More: Extreme Gradient Boost Rank-1 Adaption for Efficient Finetuning of LLMs [75.11449420928139]
  Fine-tuning Large Language Models (LLMs) has become a crucial technique for adapting pre-trained models to downstream tasks. Low-Rank Adaptation (LoRA) has emerged as a promising solution, but there exists a gap between the practical performance of low-rank adaptations and its theoretical optimum. We propose eXtreme Gradient Boosting LoRA, a novel framework that bridges this gap by leveraging the power of ensemble learning.
  arXiv  Detail & Related papers  (2024-10-25T17:07:13Z)
• Scaling Optimal LR Across Token Horizons [81.29631219839311]
  We show how optimal learning rate depends on token horizon in LLM training.<n>We also provide evidence that LLama-1 used too high LR, and estimate the performance hit from this.
  arXiv  Detail & Related papers  (2024-09-30T03:32:02Z)
• SHERL: Synthesizing High Accuracy and Efficient Memory for Resource-Limited Transfer Learning [63.93193829913252]
  We propose an innovative METL strategy called SHERL for resource-limited scenarios. In the early route, intermediate outputs are consolidated via an anti-redundancy operation. In the late route, utilizing minimal late pre-trained layers could alleviate the peak demand on memory overhead.
  arXiv  Detail & Related papers  (2024-07-10T10:22:35Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• On the Weight Dynamics of Deep Normalized Networks [5.250288418639077]
  High disparities in effective learning rates (ELRs) across layers in deep neural networks can negatively affect trainability. We formalize how these disparities evolve over time by modeling weight dynamics of networks with normalization layers. We prove that when training with any constant learning rate, ELR ratios converge to 1, despite initial gradient explosion.
  arXiv  Detail & Related papers  (2023-06-01T14:09:52Z)
• Surrogate Lagrangian Relaxation: A Path To Retrain-free Deep Neural Network Pruning [9.33753001494221]
  Network pruning is a widely used technique to reduce computation cost and model size for deep neural networks. In this paper, we develop a systematic weight-pruning optimization approach based on Surrogate Lagrangian relaxation.
  arXiv  Detail & Related papers  (2023-04-08T22:48:30Z)
• Regularized linear autoencoders recover the principal components, eventually [15.090789983727335]
  We show that when trained with proper regularization, linear autoencoders can learn the optimal representation. We show that this convergence is slow due to ill-conditioning that worsens with increasing latent dimension. We present a simple modification to the gradient descent update that greatly speeds up empirically.
  arXiv  Detail & Related papers  (2020-07-13T23:08:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.