A geometric framework for momentum-based optimizers for low-rank training

• URL: http://arxiv.org/abs/2506.17475v1
• Date: Fri, 20 Jun 2025 20:46:01 GMT
• Title: A geometric framework for momentum-based optimizers for low-rank training
• Authors: Steffen Schotthöfer, Timon Klein, Jonas Kusch,
• Abstract summary: Low-rank pre-training and fine-tuning have emerged as promising techniques for reducing the computational and storage costs of large neural networks.<n>We show that classical momentum methods can struggle to converge to a local optimum due to the geometry of the underlying optimization landscape.<n>We introduce novel training strategies derived from dynamical low-rank approximation, which explicitly account for the underlying geometric structure.
• Score: 2.389598109913754
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Low-rank pre-training and fine-tuning have recently emerged as promising techniques for reducing the computational and storage costs of large neural networks. Training low-rank parameterizations typically relies on conventional optimizers such as heavy ball momentum methods or Adam. In this work, we identify and analyze potential difficulties that these training methods encounter when used to train low-rank parameterizations of weights. In particular, we show that classical momentum methods can struggle to converge to a local optimum due to the geometry of the underlying optimization landscape. To address this, we introduce novel training strategies derived from dynamical low-rank approximation, which explicitly account for the underlying geometric structure. Our approach leverages and combines tools from dynamical low-rank approximation and momentum-based optimization to design optimizers that respect the intrinsic geometry of the parameter space. We validate our methods through numerical experiments, demonstrating faster convergence, and stronger validation metrics at given parameter budgets.

Related papers

• Exploring the Impact of Parameter Update Magnitude on Forgetting and Generalization of Continual Learning [11.882528379148141]
  The magnitude of parameter updates are considered a key factor in continual learning.<n>We unify two representative update paradigms, frozen training and generalization training.<n>Experiments on deep neural networks demonstrate that this hybrid approach outperforms standard training strategies.
  arXiv  Detail & Related papers  (2026-02-24T11:35:15Z)
• ODELoRA: Training Low-Rank Adaptation by Solving Ordinary Differential Equations [54.886931928255564]
  Low-rank adaptation (LoRA) has emerged as a widely adopted parameter-efficient fine-tuning method in deep transfer learning.<n>We propose a novel continuous-time optimization dynamic for LoRA factor matrices in the form of an ordinary differential equation (ODE)<n>We show that ODELoRA achieves stable feature learning, a property that is crucial for training deep neural networks at different scales of problem dimensionality.
  arXiv  Detail & Related papers  (2026-02-07T10:19:36Z)
• The Optimiser Hidden in Plain Sight: Training with the Loss Landscape's Induced Metric [0.0]
  We present a class of novel optimisers for training neural networks.<n>The new optimiser has a computational complexity comparable to that of Adam.<n>One variant of these optimisers can also be viewed as inducing an effective scheduled learning rate.
  arXiv  Detail & Related papers  (2025-09-03T18:00:33Z)
• Functional Neural Wavefunction Optimization [11.55213641895401]
  We propose a framework for the design and analysis of optimization algorithms in variational quantum Monte Carlo.<n>The framework translates infinite-dimensional optimization dynamics into tractable parameter-space algorithms.<n>We validate our framework with numerical experiments demonstrating its practical relevance.
  arXiv  Detail & Related papers  (2025-07-14T22:07:38Z)
• Improving Learning to Optimize Using Parameter Symmetries [16.76912881772023]
  We analyze a learning-to-optimize (L2O) algorithm that exploits parameter space symmetry to enhance efficiency.<n>Our results highlight the potential of leveraging neural network parameter space symmetry to advance meta-optimization.
  arXiv  Detail & Related papers  (2025-04-21T19:03:23Z)
• Training Deep Learning Models with Norm-Constrained LMOs [56.00317694850397]
  We propose a new family of algorithms that uses the linear minimization oracle (LMO) to adapt to the geometry of the problem.<n>We demonstrate significant speedups on nanoGPT training using our algorithm, Scion, without any reliance on Adam.
  arXiv  Detail & Related papers  (2025-02-11T13:10:34Z)
• AdaRankGrad: Adaptive Gradient-Rank and Moments for Memory-Efficient LLMs Training and Fine-Tuning [9.51289606759621]
  Training and fine-tuning large language models (LLMs) come with challenges related to memory and computational requirements.<n>Various techniques have been developed to tackle these challenges, such as low-rank adaptation (LoRA)<n>We introduce a new method inspired by a phenomenon we formally prove: as training progresses, the rank of the estimated gradient gradually decreases.
  arXiv  Detail & Related papers  (2024-10-23T13:53:26Z)
• Hallmarks of Optimization Trajectories in Neural Networks: Directional Exploration and Redundancy [75.15685966213832]
  We analyze the rich directional structure of optimization trajectories represented by their pointwise parameters. We show that training only scalar batchnorm parameters some while into training matches the performance of training the entire network.
  arXiv  Detail & Related papers  (2024-03-12T07:32:47Z)
• Gradient-free neural topology optimization [0.0]
  gradient-free algorithms require many more iterations to converge when compared to gradient-based algorithms. This has made them unviable for topology optimization due to the high computational cost per iteration and high dimensionality of these problems. We propose a pre-trained neural reparameterization strategy that leads to at least one order of magnitude decrease in iteration count when optimizing the designs in latent space.
  arXiv  Detail & Related papers  (2024-03-07T23:00:49Z)
• Multiplicative update rules for accelerating deep learning training and increasing robustness [69.90473612073767]
  We propose an optimization framework that fits to a wide range of machine learning algorithms and enables one to apply alternative update rules. We claim that the proposed framework accelerates training, while leading to more robust models in contrast to traditionally used additive update rule.
  arXiv  Detail & Related papers  (2023-07-14T06:44:43Z)
• Transformer-Based Learned Optimization [37.84626515073609]
  We propose a new approach to learned optimization where we represent the computation's update step using a neural network. Our innovation is a new neural network architecture inspired by the classic BFGS algorithm. We demonstrate the advantages of our approach on a benchmark composed of objective functions traditionally used for the evaluation of optimization algorithms.
  arXiv  Detail & Related papers  (2022-12-02T09:47:08Z)
• Powerpropagation: A sparsity inducing weight reparameterisation [65.85142037667065]
  We introduce Powerpropagation, a new weight- parameterisation for neural networks that leads to inherently sparse models. Models trained in this manner exhibit similar performance, but have a distribution with markedly higher density at zero, allowing more parameters to be pruned safely. Here, we combine Powerpropagation with a traditional weight-pruning technique as well as recent state-of-the-art sparse-to-sparse algorithms, showing superior performance on the ImageNet benchmark.
  arXiv  Detail & Related papers  (2021-10-01T10:03:57Z)
• Optimization on manifolds: A symplectic approach [127.54402681305629]
  We propose a dissipative extension of Dirac's theory of constrained Hamiltonian systems as a general framework for solving optimization problems. Our class of (accelerated) algorithms are not only simple and efficient but also applicable to a broad range of contexts.
  arXiv  Detail & Related papers  (2021-07-23T13:43:34Z)
• An AI-Assisted Design Method for Topology Optimization Without Pre-Optimized Training Data [68.8204255655161]
  An AI-assisted design method based on topology optimization is presented, which is able to obtain optimized designs in a direct way. Designs are provided by an artificial neural network, the predictor, on the basis of boundary conditions and degree of filling as input data.
  arXiv  Detail & Related papers  (2020-12-11T14:33:27Z)
• Automatically Learning Compact Quality-aware Surrogates for Optimization Problems [55.94450542785096]
  Solving optimization problems with unknown parameters requires learning a predictive model to predict the values of the unknown parameters and then solving the problem using these values. Recent work has shown that including the optimization problem as a layer in a complex training model pipeline results in predictions of iteration of unobserved decision making. We show that we can improve solution quality by learning a low-dimensional surrogate model of a large optimization problem.
  arXiv  Detail & Related papers  (2020-06-18T19:11:54Z)
• Extrapolation for Large-batch Training in Deep Learning [72.61259487233214]
  We show that a host of variations can be covered in a unified framework that we propose. We prove the convergence of this novel scheme and rigorously evaluate its empirical performance on ResNet, LSTM, and Transformer.
  arXiv  Detail & Related papers  (2020-06-10T08:22:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.