Does SGD really happen in tiny subspaces?

• URL: http://arxiv.org/abs/2405.16002v3
• Date: Tue, 11 Mar 2025 01:07:24 GMT
• Title: Does SGD really happen in tiny subspaces?
• Abstract summary: Recent studies have revealed that, along the training trajectory, the gradient aligns with a low-rank top eigenspace of the training loss Hessian.<n>This paper explores whether neural networks can be trained within the dominant subspace, which could lead to more efficient training methods.
• Score: 18.283839252425803
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Understanding the training dynamics of deep neural networks is challenging due to their high-dimensional nature and intricate loss landscapes. Recent studies have revealed that, along the training trajectory, the gradient approximately aligns with a low-rank top eigenspace of the training loss Hessian, referred to as the dominant subspace. Given this alignment, this paper explores whether neural networks can be trained within the dominant subspace, which, if feasible, could lead to more efficient training methods. Our primary observation is that when the SGD update is projected onto the dominant subspace, the training loss does not decrease further. This suggests that the observed alignment between the gradient and the dominant subspace is spurious. Surprisingly, projecting out the dominant subspace proves to be just as effective as the original update, despite removing the majority of the original update component. We observe similar behavior across practical setups, including the large learning rate regime (also known as Edge of Stability), Sharpness-Aware Minimization, momentum, and adaptive optimizers. We discuss the main causes and implications of this spurious alignment, shedding light on the dynamics of neural network training.

Related papers

• KeepLoRA++: Continual Learning with Layer-Scaled Residual Gradient Adaptation [67.23448686598414]
  Continual learning for pre-trained vision-language models requires balancing three competing objectives.<n>This paper presents KeepLoRA++, balancing these objectives through a unified dual-dimensional knowledge retention mechanism.
  arXiv  Detail & Related papers  (2026-06-15T05:57:59Z)
• Randomized Gradient Subspaces for Efficient Large Language Model Training [4.398170461093705]
  Training large language models (LLMs) is often bottlenecked by extreme memory demands, with states dominating the footprint.<n>Recent works mitigate this cost by projecting gradients into low-dimensional subspaces using sophisticated update strategies.<n>We find that while a small subspace captures most gradient energy, a significant portion still resides in the residual bulk.<n>We introduce a suite of randomized algorithms, GrassWalk and GrassJump, which exploit subspace and achieve state-of-the-art memory savings while improving performance on LLaMA-1B and LLaMA-7B pretraining.
  arXiv  Detail & Related papers  (2025-10-02T10:35:38Z)
• SplitLoRA: Balancing Stability and Plasticity in Continual Learning Through Gradient Space Splitting [68.00007494819798]
  Continual learning requires a model to learn multiple tasks in sequence while maintaining both stability:preserving knowledge from previously learned tasks, and plasticity:effectively learning new tasks.<n> Gradient projection has emerged as an effective and popular paradigm in CL, where it partitions the gradient space of previously learned tasks into two subspaces.<n>New tasks are learned effectively within the minor subspace, thereby reducing interference with previously acquired knowledge.<n>Existing Gradient Projection methods struggle to achieve an optimal balance between plasticity and stability, as it is hard to appropriately partition the gradient space.
  arXiv  Detail & Related papers  (2025-05-28T13:57:56Z)
• Take A Shortcut Back: Mitigating the Gradient Vanishing for Training Spiking Neural Networks [15.691263438655842]
  Spiking Neural Network (SNN) is a biologically inspired neural network infrastructure that has recently garnered significant attention. Training an SNN directly poses a challenge due to the undefined gradient of the firing spike process. We propose a shortcut back-propagation method in our paper, which advocates for transmitting the gradient directly from the loss to the shallow layers.
  arXiv  Detail & Related papers  (2024-01-09T10:54:41Z)
• Are GATs Out of Balance? [73.2500577189791]
  We study the Graph Attention Network (GAT) in which a node's neighborhood aggregation is weighted by parameterized attention coefficients. Our main theorem serves as a stepping stone to studying the learning dynamics of positive homogeneous models with attention mechanisms.
  arXiv  Detail & Related papers  (2023-10-11T06:53:05Z)
• The Law of Parsimony in Gradient Descent for Learning Deep Linear Networks [34.85235641812005]
  We reveal a surprising "law of parsimony" in the learning dynamics when the data possesses low-dimensional structures. This simplicity in learning dynamics could have significant implications for both efficient training and a better understanding of deep networks.
  arXiv  Detail & Related papers  (2023-06-01T21:24:53Z)
• Sharpness-Aware Minimization Leads to Low-Rank Features [49.64754316927016]
  Sharpness-aware minimization (SAM) is a recently proposed method that minimizes the training loss of a neural network. We show that SAM reduces the feature rank which happens at different layers of a neural network. We confirm this effect theoretically and check that it can also occur in deep networks.
  arXiv  Detail & Related papers  (2023-05-25T17:46:53Z)
• Adversarial Examples Exist in Two-Layer ReLU Networks for Low Dimensional Linear Subspaces [24.43191276129614]
  We show that standard methods lead to non-robust neural networks. We show that decreasing the scale of the training algorithm, or adding $L$ regularization, can make the trained network more robust to adversarial perturbations.
  arXiv  Detail & Related papers  (2023-03-01T19:10:05Z)
• SGD with Large Step Sizes Learns Sparse Features [22.959258640051342]
  We showcase important features of the dynamics of the Gradient Descent (SGD) in the training of neural networks. We show that the longer large step sizes keep SGD high in the loss landscape, the better the implicit regularization can operate and find sparse representations.
  arXiv  Detail & Related papers  (2022-10-11T11:00:04Z)
• Slimmable Networks for Contrastive Self-supervised Learning [69.9454691873866]
  Self-supervised learning makes significant progress in pre-training large models, but struggles with small models. We introduce another one-stage solution to obtain pre-trained small models without the need for extra teachers. A slimmable network consists of a full network and several weight-sharing sub-networks, which can be pre-trained once to obtain various networks.
  arXiv  Detail & Related papers  (2022-09-30T15:15:05Z)
• Minimizing Control for Credit Assignment with Strong Feedback [65.59995261310529]
  Current methods for gradient-based credit assignment in deep neural networks need infinitesimally small feedback signals. We combine strong feedback influences on neural activity with gradient-based learning and show that this naturally leads to a novel view on neural network optimization. We show that the use of strong feedback in DFC allows learning forward and feedback connections simultaneously, using a learning rule fully local in space and time.
  arXiv  Detail & Related papers  (2022-04-14T22:06:21Z)
• There is a Singularity in the Loss Landscape [0.0]
  We show that as the size of the dataset increases, a point forms where the magnitude of the gradient of the loss becomes unbounded. This singularity explains a variety of phenomena recently observed in the Hessian of neural network loss functions.
  arXiv  Detail & Related papers  (2022-01-12T01:50:06Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Exploring the Common Principal Subspace of Deep Features in Neural Networks [50.37178960258464]
  We find that different Deep Neural Networks (DNNs) trained with the same dataset share a common principal subspace in latent spaces. Specifically, we design a new metric $mathcalP$-vector to represent the principal subspace of deep features learned in a DNN. Small angles (with cosine close to $1.0$) have been found in the comparisons between any two DNNs trained with different algorithms/architectures.
  arXiv  Detail & Related papers  (2021-10-06T15:48:32Z)
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability [94.4070247697549]
  Full-batch gradient descent on neural network training objectives operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / text(step size)$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales.
  arXiv  Detail & Related papers  (2021-02-26T22:08:19Z)
• Align, then memorise: the dynamics of learning with feedback alignment [12.587037358391418]
  Direct Feedback Alignment (DFA) is an efficient alternative to the ubiquitous backpropagation algorithm for training deep neural networks. DFA successfully trains state-of-the-art models such as Transformers, but it notoriously fails to train convolutional networks. Here, we propose a theory for the success of DFA.
  arXiv  Detail & Related papers  (2020-11-24T22:21:27Z)
• Improving Performance in Reinforcement Learning by Breaking Generalization in Neural Networks [5.273501657421096]
  We show how online NN training and interference interact in reinforcement learning. We find that simply re-mapping the input observations to a high-dimensional space improves learning speed and parameter sensitivity. We provide a simple approach to NN training that is easy to implement, and requires little additional computation.
  arXiv  Detail & Related papers  (2020-03-16T19:21:08Z)
• The Break-Even Point on Optimization Trajectories of Deep Neural Networks [64.7563588124004]
  We argue for the existence of the "break-even" point on this trajectory. We show that using a large learning rate in the initial phase of training reduces the variance of the gradient. We also show that using a low learning rate results in bad conditioning of the loss surface even for a neural network with batch normalization layers.
  arXiv  Detail & Related papers  (2020-02-21T22:55:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.