Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization

• URL: http://arxiv.org/abs/2606.30813v1
• Date: Mon, 29 Jun 2026 18:37:34 GMT
• Title: Gradient Smoothing: Coupling Layer-wise Updates for Improved Optimization
• Abstract summary: We introduce emphDepth-wise Gradient Augmentation, a general optimization paradigm in which the update applied to each layer is obtained by transforming the collection of block-wise updates along the depth dimension.<n>Within this framework, we study emphGradient Smoothing, a family of depth-wise smoothing methods, and instantiate it with a simple local emphWindow Smoothing operator.
• Score: 15.73948621729859
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep neural networks with repeated architectural blocks, such as transformers, often exhibit structured relationships across layers that emerge during training. Motivated by this observation, we introduce \emph{Depth-wise Gradient Augmentation}, a general optimization paradigm in which the update applied to each layer is obtained by transforming the collection of block-wise optimizer updates along the depth dimension. Within this framework, we study \emph{Gradient Smoothing}, a family of depth-wise smoothing methods, and instantiate it with a simple local \emph{Window Smoothing} operator. The resulting method operates directly on block-wise updates produced by arbitrary base optimizers (e.g., SGD, Adam, Muon), incurs minimal computational overhead, and is compatible with existing optimization pipelines. We evaluate Gradient Smoothing across a diverse set of architectures and training regimes, including language model pretraining, RL post-training of LLMs for reasoning, diffusion modeling, and image classification with Vision Transformers. Across these settings, Gradient Smoothing consistently improves optimization and generalization performance without modifying model architectures or training objectives. We further show that it promotes more structured representation evolution across depth, consistent with its interpretation as a structured depth-wise preconditioning method. Together, these results establish Depth-wise Gradient Augmentation as a promising framework for exploiting cross-depth structure in optimization and demonstrate Gradient Smoothing as a simple and broadly applicable instantiation.

Related papers

• Neural Network Optimization Reimagined: Decoupled Techniques for Scratch and Fine-Tuning [49.751529745537546]
  We propose DualOpt, a novel approach that decouples optimization techniques specifically tailored for training from scratch.<n>For training from scratch, we introduce real-time layer-wise weight decay, designed to enhance both convergence and generalization.<n>We extend the layer-wise weight decay to dynamically adjust the rollback levels across layers, adapting to the varying demands of different downstream tasks.
  arXiv  Detail & Related papers  (2026-04-21T06:27:18Z)
• CogniEdit: Dense Gradient Flow Optimization for Fine-Grained Image Editing [88.9067184995168]
  We propose a unified framework CogniEdit, combining multi-modal reasoning with dense reward optimization.<n>Our method achieves state-of-the-art performance in balancing fine-grained instruction following with visual quality and editability preservation.
  arXiv  Detail & Related papers  (2025-12-15T12:36:50Z)
• IDAP++: Advancing Divergence-Based Pruning via Filter-Level and Layer-Level Optimization [38.16787019088681]
  This paper presents a novel approach to neural network compression that addresses redundancy at both the filter and architectural levels.<n>We build on the concept of tensor flow divergence, which quantifies how information is transformed across network layers.
  arXiv  Detail & Related papers  (2025-11-25T10:02:21Z)
• TreeLoRA: Efficient Continual Learning via Layer-Wise LoRAs Guided by a Hierarchical Gradient-Similarity Tree [52.44403214958304]
  In this paper, we introduce TreeLoRA, a novel approach that constructs layer-wise adapters by leveraging hierarchical gradient similarity.<n>To reduce the computational burden of task similarity estimation, we employ bandit techniques to develop an algorithm based on lower confidence bounds.<n> experiments on both vision transformers (ViTs) and large language models (LLMs) demonstrate the effectiveness and efficiency of our approach.
  arXiv  Detail & Related papers  (2025-06-12T05:25:35Z)
• Contextually Entangled Gradient Mapping for Optimized LLM Comprehension [0.0]
  Entually Entangled Gradient Mapping (CEGM) introduces a new approach to gradient optimization.<n>It treats gradients as dynamic carriers of contextual dependencies rather than isolated numerical entities.<n>The proposed methodology bridges critical gaps in existing optimization strategies.
  arXiv  Detail & Related papers  (2025-01-28T11:50:35Z)
• Context-Aware Neural Gradient Mapping for Fine-Grained Instruction Processing [0.0]
  This paper introduces a dynamic gradient adjustment mechanism, incorporating contextual embeddings directly into the optimization process.<n>The proposed framework consistently outperforms baseline models across various metrics, including accuracy, robustness to noise, and computational efficiency.<n>The integration of context-specific embeddings allows for a more complex understanding of language, thereby improving the model's ability to handle diverse linguistic phenomena.
  arXiv  Detail & Related papers  (2025-01-24T21:49:24Z)
• CaAdam: Improving Adam optimizer using connection aware methods [0.0]
  We introduce a new method inspired by Adam that enhances convergence speed and achieves better loss function minima. Traditional proxies, including Adam, apply uniform or globally adjusted learning rates across neural networks without considering their architectural specifics. Our algorithm, CaAdam, explores this overlooked area by introducing connection-aware optimization through carefully designed of architectural information.
  arXiv  Detail & Related papers  (2024-10-31T17:59:46Z)
• Multiplicative update rules for accelerating deep learning training and increasing robustness [69.90473612073767]
  We propose an optimization framework that fits to a wide range of machine learning algorithms and enables one to apply alternative update rules. We claim that the proposed framework accelerates training, while leading to more robust models in contrast to traditionally used additive update rule.
  arXiv  Detail & Related papers  (2023-07-14T06:44:43Z)
• GradMax: Growing Neural Networks using Gradient Information [22.986063120002353]
  We present a method that adds new neurons during training without impacting what is already learned, while improving the training dynamics. We call this technique Gradient Maximizing Growth (GradMax) and demonstrate its effectiveness in variety of vision tasks and architectures.
  arXiv  Detail & Related papers  (2022-01-13T18:30:18Z)
• Dynamically Grown Generative Adversarial Networks [111.43128389995341]
  We propose a method to dynamically grow a GAN during training, optimizing the network architecture and its parameters together with automation. The method embeds architecture search techniques as an interleaving step with gradient-based training to periodically seek the optimal architecture-growing strategy for the generator and discriminator.
  arXiv  Detail & Related papers  (2021-06-16T01:25:51Z)
• Dynamic Hierarchical Mimicking Towards Consistent Optimization Objectives [73.15276998621582]
  We propose a generic feature learning mechanism to advance CNN training with enhanced generalization ability. Partially inspired by DSN, we fork delicately designed side branches from the intermediate layers of a given neural network. Experiments on both category and instance recognition tasks demonstrate the substantial improvements of our proposed method.
  arXiv  Detail & Related papers  (2020-03-24T09:56:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.