One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs

• URL: http://arxiv.org/abs/2605.22297v3
• Date: Wed, 27 May 2026 10:11:58 GMT
• Title: One LR Doesn't Fit All: Heavy-Tail Guided Layerwise Learning Rates for LLMs
• Abstract summary: Layerwise Learning Rate (LLR) is an adaptive scheme that assigns distinct learning rates to individual Transformer layers.<n>LLR achieves up to 1.5x training speedup and consistently outperforms uniform-learning-rate baselines.
• Score: 19.49856488618013
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Learning rate configuration is a fundamental aspect of modern deep learning. The prevailing practice of applying a uniform learning rate across all layers overlooks the structural heterogeneity of Transformers, potentially limiting their effectiveness as the backbone of Large Language Models (LLMs). In this paper, we introduce Layerwise Learning Rate (LLR), an adaptive scheme that assigns distinct learning rates to individual Transformer layers. Our method is grounded in Heavy-Tailed Self-Regularization (HT-SR) theory, which characterizes the empirical spectral density (ESD) of weight correlation matrices to quantify heavy-tailedness. Layers with weaker heavy-tailedness are assigned larger learning rates to accelerate training, while layers with stronger heavy-tailedness receive smaller learning rates. By tailoring learning rates in this manner, LLR promotes more balanced training across layers, leading to faster convergence and improved generalization. Extensive experiments across architectures ranging from LLaMA to GPT-nano, optimizers including AdamW and Muon, and model scales from 60M to 3B parameters with up to 100B training tokens demonstrate the effectiveness of LLR. LLR achieves up to 1.5x training speedup and consistently outperforms uniform-learning-rate baselines. In particular, it improves the average zero-shot accuracy of 1B models from 47.09% to 49.02%, and that of 3B models from 48.58% to 50.61%. A key advantage of LLR is its low tuning overhead: it can transfer nearly optimal learning-rate settings directly from the uniform baseline. Code is available at https://github.com/hed-ucas/Layer-wise-Learning-Rate.

Related papers

• Parameter-Efficient Fine-Tuning with Learnable Rank [30.391374552207008]
  Low-Rank Adaptation (LoRA) is a popular parameter-efficient finetuning (PEFT) method that restricts weight updates to low-rank adapters.<n>We introduce *Learnable Rank LoRA (LR-LoRA)*, a PEFT method in which the adapter rank is learned during the training process.
  arXiv  Detail & Related papers  (2026-06-03T00:57:36Z)
• Learning Rate Matters: Vanilla LoRA May Suffice for LLM Fine-tuning [48.66442009036754]
  Low-Rank Adaptation (LoRA) is the prevailing approach for efficient large language model fine-tuning.<n>In this work, we re-evaluate four representative LoRA variants alongside vanilla LoRA.<n>We find that different LoRA methods favor distinct learning rate ranges.
  arXiv  Detail & Related papers  (2026-02-04T19:36:20Z)
• Data Distribution as a Lever for Guiding Optimizers Toward Superior Generalization in LLMs [60.68927774057402]
  We show, for the first time, that a lower simplicity bias induces a better generalization.<n>Motivated by this insight, we demonstrate that the training data distribution by upsampling or augmenting examples learned later in training similarly reduces SB and leads to improved generalization.<n>Our strategy improves the performance of multiple language models including Phi2-2.7B, Llama3.2-1B, Gemma3-1B-PT, Qwen3-0.6B-Base-achieving relative accuracy gains up to 18% when fine-tuned with AdamW and Muon.
  arXiv  Detail & Related papers  (2026-01-31T07:40:36Z)
• Accelerating Reinforcement Learning Algorithms Convergence using Pre-trained Large Language Models as Tutors With Advice Reusing [5.414308305392762]
  Large Language Models (LLMs) as tutors in a student-teacher architecture withReinforcement Learning (RL) algorithms.<n>Our results demonstrate that LLM tutoring significantly accelerates RL convergence while maintaining comparable optimal performance.<n>Advice reuse mechanism shows a further improvement in training duration but also results in less stable convergence dynamics.
  arXiv  Detail & Related papers  (2025-09-10T07:08:04Z)
• Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models [49.911784762244814]
  TraceRL is a trajectory-aware reinforcement learning framework for diffusion language models (DLMs)<n>We derive a series of state-of-the-art diffusion language models, namely TraDo.<n>TraDo-8B-Instruct achieves relative accuracy improvements of 6.1% over Qwen2.5-7B-Instruct and 51.3% over Llama3.1-8B-Instruct on mathematical reasoning benchmarks.
  arXiv  Detail & Related papers  (2025-09-08T17:58:06Z)
• Shuffle-R1: Efficient RL framework for Multimodal Large Language Models via Data-centric Dynamic Shuffle [65.14124923451077]
  Reinforcement learning (RL) has emerged as an effective post-training paradigm for enhancing the reasoning capabilities of multimodal large language model (MLLM)<n>However, current RL pipelines often suffer from training inefficiencies caused by two underexplored issues: Advantage Collapsing and Rollout Silencing.<n>We propose Shuffle-R1, a simple yet principled framework that improves RL fine-tuning efficiency by dynamically restructuring trajectory sampling and batch composition.
  arXiv  Detail & Related papers  (2025-08-07T17:53:47Z)
• Decoupled Relative Learning Rate Schedules [4.34286535607654]
  We introduce a novel approach for optimizing LLM training by adjusting learning rates across weights of different components in Transformer models.<n>Our introduced relative learning rates, RLRS, method accelerates the training process by up to $23%$.
  arXiv  Detail & Related papers  (2025-07-04T12:23:45Z)
• Boosting Deep Ensembles with Learning Rate Tuning [1.6021932740447968]
  Learning Rate (LR) has a high impact on deep learning training performance. This paper presents a novel framework, LREnsemble, to leverage effective learning rate tuning to boost deep ensemble performance.
  arXiv  Detail & Related papers  (2024-10-10T02:59:38Z)
• Fisher Information-based Efficient Curriculum Federated Learning with Large Language Models [43.26028399395612]
  We propose a Fisher Information-based Efficient Curriculum Federated Learning framework (FibecFed) with two novel methods. First, we propose a fisher information-based method to adaptively sample data within each device to improve the effectiveness of the FL fine-tuning process. Second, we dynamically select the proper layers for global aggregation and sparse parameters for local update with LoRA.
  arXiv  Detail & Related papers  (2024-09-30T18:12:18Z)
• Scaling Optimal LR Across Token Horizons [81.29631219839311]
  We show how optimal learning rate depends on token horizon in LLM training.<n>We also provide evidence that LLama-1 used too high LR, and estimate the performance hit from this.
  arXiv  Detail & Related papers  (2024-09-30T03:32:02Z)
• Outlier-weighed Layerwise Sampling for LLM Fine-tuning [18.102930806071978]
  Outlier-weighed Layerwise Sampling is a memory-efficient fine-tuning approach.<n>We show that OWS consistently outperforms baseline approaches, including full fine-tuning.
  arXiv  Detail & Related papers  (2024-05-28T17:22:22Z)
• PRILoRA: Pruned and Rank-Increasing Low-Rank Adaptation [65.268245109828]
  We introduce PRILoRA, which linearly allocates a different rank for each layer, in an increasing manner, and performs pruning throughout the training process. We validate the effectiveness of PRILoRA through extensive experiments on eight GLUE benchmarks, setting a new state of the art.
  arXiv  Detail & Related papers  (2024-01-20T20:25:17Z)
• On the Weight Dynamics of Deep Normalized Networks [5.250288418639077]
  High disparities in effective learning rates (ELRs) across layers in deep neural networks can negatively affect trainability. We formalize how these disparities evolve over time by modeling weight dynamics of networks with normalization layers. We prove that when training with any constant learning rate, ELR ratios converge to 1, despite initial gradient explosion.
  arXiv  Detail & Related papers  (2023-06-01T14:09:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.