Factored Levenberg-Marquardt for Diffeomorphic Image Registration: An efficient optimizer for FireANTs

• URL: http://arxiv.org/abs/2603.19371v1
• Date: Thu, 19 Mar 2026 18:04:42 GMT
• Title: Factored Levenberg-Marquardt for Diffeomorphic Image Registration: An efficient optimizer for FireANTs
• Abstract summary: We propose a modified Levenberg-Marquardt (LM) that requires only a single damping parameter as a state, that is adaptively tuned using a trust region approach.<n>The resulting reduces memory by up to 24.6% for large volumes, and retaining performance across all four datasets.
• Score: 20.35907245543535
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: FireANTs introduced a novel Eulerian descent method for plug-and-play behavior with arbitrary optimizers adapted for diffeomorphic image registration as a test-time optimization problem, with a GPU-accelerated implementation. FireANTs uses Adam as its default optimizer for fast and more robust optimization. However, Adam requires storing state variables (i.e. momentum and squared-momentum estimates), each of which can consume significant memory, prohibiting its use for significantly large images. In this work, we propose a modified Levenberg-Marquardt (LM) optimizer that requires only a single scalar damping parameter as optimizer state, that is adaptively tuned using a trust region approach. The resulting optimizer reduces memory by up to 24.6% for large volumes, and retaining performance across all four datasets. A single hyperparameter configuration tuned on brain MRI transfers without modification to lung CT and cross-modal abdominal registration, matching or outperforming Adam on three of four benchmarks. We also perform ablations on the effectiveness of using Metropolis-Hastings style rejection step to prevent updates that worsen the loss function.

Related papers

• Reducing Per-Sample Harm in Stochastic Optimization [1.5229257192293202]
  Moderns combine gradients from the current mini-batch with historical optimization state, such as momentum or adaptive moments.<n>We term this effect harm and formalize the parameter update as an optimization problem.<n>Because the exact formulation is intractable, we introduce a highly efficient proxy.
  arXiv  Detail & Related papers  (2026-06-28T18:01:27Z)
• Low-rank Momentum Factorization for Memory Efficient Training [13.464518325870444]
  Momentum Factorized (MoFaSGD) maintains a dynamically updated low-rank SVD representation of the first-order momentum.<n>We demonstrate MoFaSGD's effectiveness on large language model benchmarks, achieving a competitive trade-off between memory reduction (e.g. LoRA) and performance.
  arXiv  Detail & Related papers  (2025-07-10T18:04:52Z)
• A Minimalist Optimizer Design for LLM Pretraining [31.996047271119156]
  Training large language models typically relies on adaptives such as Adam.<n>Recent works such as GaLore Fira, and APOLLO have proposed state-compressed variants to reduce memory consumption.<n>In this work, we investigate what is the minimal amount of state that is truly necessary to retain state-of-the-art performance in LLM pretraining.
  arXiv  Detail & Related papers  (2025-06-20T00:10:35Z)
• Pushing the Limits of Low-Bit Optimizers: A Focus on EMA Dynamics [64.62231094774211]
  Statefuls (e.g., Adam) maintain auxiliary information even 2x the model size in order to achieve optimal convergence.<n>SOLO enables Adam-styles to maintain quantized states with precision as low as 3 bits, or even 2 bits.<n>SOLO can thus be seamlessly applied to Adam-styles, leading to substantial memory savings with minimal accuracy loss.
  arXiv  Detail & Related papers  (2025-05-01T06:47:45Z)
• COSMOS: A Hybrid Adaptive Optimizer for Memory-Efficient Training of LLMs [77.79640601822341]
  Large Language Models (LLMs) have demonstrated remarkable success across various domains.<n>Their optimization remains a significant challenge due to the complex and high-dimensional loss landscapes they inhabit.
  arXiv  Detail & Related papers  (2025-02-24T18:42:19Z)
• APOLLO: SGD-like Memory, AdamW-level Performance [61.53444035835778]
  Large language models (LLMs) are notoriously memory-intensive during training.<n>Various memory-efficient Scals have been proposed to reduce memory usage.<n>They face critical challenges: (i) costly SVD operations; (ii) significant performance trade-offs compared to AdamW; and (iii) still substantial memory overhead to maintain competitive performance.
  arXiv  Detail & Related papers  (2024-12-06T18:55:34Z)
• Simultaneous Computation and Memory Efficient Zeroth-Order Optimizer for Fine-Tuning Large Language Models [33.911521719528686]
  Fine-tuning is powerful for adapting large language models to downstream tasks, but it often results in huge memory usages. A promising approach is using Zeroth-Order (ZO) gradients, which estimates to replace First-Order (FO) gradients. We introduce a novel layer-wise sparse computation and memory efficient ZO, named LeZO.
  arXiv  Detail & Related papers  (2024-10-13T12:47:37Z)
• 3DGS-LM: Faster Gaussian-Splatting Optimization with Levenberg-Marquardt [65.25603275491544]
  We present 3DGS-LM, a new method that accelerates the reconstruction of 3D Gaussian Splatting (3DGS)<n>Our method is 20% faster than the original 3DGS while obtaining the same reconstruction quality optimization.
  arXiv  Detail & Related papers  (2024-09-19T16:31:44Z)
• Sparse MeZO: Less Parameters for Better Performance in Zeroth-Order LLM Fine-Tuning [67.44661423463927]
  This paper introduces Sparse MeZO, a memory-efficient zeroth-order optimization approach that applies ZO only to a carefully chosen subset of parameters. We show that Sparse-MeZO consistently improves both performance and convergence speed over MeZO without any overhead.
  arXiv  Detail & Related papers  (2024-02-24T07:22:04Z)
• AdaLomo: Low-memory Optimization with Adaptive Learning Rate [59.64965955386855]
  We introduce low-memory optimization with adaptive learning rate (AdaLomo) for large language models. AdaLomo results on par with AdamW, while significantly reducing memory requirements, thereby lowering the hardware barrier to training large language models.
  arXiv  Detail & Related papers  (2023-10-16T09:04:28Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.