MAdam: Metric-Aware Multi-Objective Adam

• URL: http://arxiv.org/abs/2606.03904v1
• Date: Tue, 02 Jun 2026 17:00:15 GMT
• Title: MAdam: Metric-Aware Multi-Objective Adam
• Abstract summary: We show two systematic gaps between the solver's intent and the coupling's execution.<n>We introduce textbfMAdam (Metric-Aware Multi-Objective Adam), a drop-in wrapper that leaves both solver and unchanged.<n>MAdam consistently improves over Adam for every solver family.
• Score: 12.820085175292377
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Multi-objective optimization (MOO) underlies many machine learning problems, yet MOO solvers across the loss-balancing, gradient-balancing, and Pareto-based families almost universally hand their reconciled directions to Adam~\cite{kingma2015adam}. We show this coupling introduces two systematic gaps between the solver's intent and the optimizer's execution. The first is a \emph{weighting mismatch}: Adam's second-moment denominator entangles the time-varying preference vector with gradient statistics, marginalizing the preference into a history average and collapsing distinct Pareto trade-offs toward a near-uniform mixture. The second is a \emph{geometric mismatch}: Adam's adaptive metric distorts the Euclidean geometry MOO solvers assume, turning aligned objectives into apparent conflicts. To resolve both jointly, we introduce \textbf{MAdam} (Metric-Aware Multi-Objective Adam), a drop-in wrapper that leaves both solver and optimizer unchanged. MAdam preconditions the reconciled direction by the preference-conditioned curvature of the scalarized objective; on this whitened input, Adam's second moment collapses to identity, so the realized update is governed by the preference-conditioned metric. Across multi-task learning, Pareto-front recovery, physics-informed neural networks, and medical imaging, MAdam consistently improves over Adam for every solver family.

Related papers

• On the Principles Behind Neural Network Optimizers [4.4366291058606535]
  This thesis develops a principled grounding for Adam and motivates new principled structures.<n>We find that the Hessian structure evolves toward a near-block-diagonal preconditioner along training.<n>We prove that this structure makes Adam's diagonal preconditioner effective.
  arXiv  Detail & Related papers  (2026-08-17T16:07:25Z)
• Adam Converges in Nonsmooth Nonconvex Optimization [3.8357180714081327]
  Adam is one of the widely implemented and influential modern bias Conversions.<n>We present the first finite-time analysis for the rate of Adam, with the iTe step and without further modifications.
  arXiv  Detail & Related papers  (2026-06-21T04:00:35Z)
• Combining Adam and its Inverse Counterpart to Enhance Generalization of Deep Learning Optimizers [57.049014152026864]
  In the training of neural networks, adaptive moment estimation (Adam) typically converges fast but exhibits suboptimal generalization performance.<n>To enhance its ability to find flat minima, we propose its new variant named inverse Adam (InvAdam)<n>InvAdam computes element-wise multiplication of the first-order and second-order moments, while Adam computes the element-wise division of these two moments.
  arXiv  Detail & Related papers  (2026-03-07T09:15:30Z)
• Conda: Column-Normalized Adam for Training Large Language Models Faster [70.66067959375748]
  Column-Normalized Adam (Conda) is a novel approach to large language models (LLMs)<n>Conda projects updates into a subspace and applies column-wise second moment normalization based on the projected gradients.<n>Experiments on the LLaMA and GPT-2 series show that Conda consistently outperforms AdamW, Muon, and other baselines in pre-training.
  arXiv  Detail & Related papers  (2025-09-29T02:58:19Z)
• UAdam: Unified Adam-Type Algorithmic Framework for Non-Convex Stochastic Optimization [20.399244578926474]
  We introduce a unified framework for Adam-type algorithms (called UAdam) This is equipped with a general form of the second-order moment, such as NAdamBound, AdaFom, and Adan. We show that UAdam converges to the neighborhood of stationary points with the rate of $mathcalO (1/T)$.
  arXiv  Detail & Related papers  (2023-05-09T13:07:03Z)
• Provable Adaptivity of Adam under Non-uniform Smoothness [79.25087082434975]
  Adam is widely adopted in practical applications due to its fast convergence. Existing convergence analyses for Adam rely on the bounded smoothness assumption. This paper studies the convergence of randomly reshuffled Adam with diminishing learning rate.
  arXiv  Detail & Related papers  (2022-08-21T14:57:47Z)
• Efficient-Adam: Communication-Efficient Distributed Adam [28.287237692902476]
  We present a novel communication complexity. $efficient distributed Adam model. Two-way quantization to reduce the cost between the server and workers.
  arXiv  Detail & Related papers  (2022-05-28T16:17:52Z)
• Understanding AdamW through Proximal Methods and Scale-Freeness [57.47324825501137]
  Adam is a generalization of the $ell$ regularizer Adam-$ell$. AdamW decouples the gradient of Adam-$ell$ from the update rule of Adam-$ell$. We show that AdamW exhibits an advantage over Adam-$ell$ and the degree to which we expect the gradients of the network to exhibit multiple scales.
  arXiv  Detail & Related papers  (2022-01-31T21:00:55Z)
• A Novel Convergence Analysis for Algorithms of the Adam Family [105.22760323075008]
  We present a generic proof of convergence for a family of Adam-style methods including Adam, AMSGrad, Adabound, etc. Our analysis is so simple and generic that it can be leveraged to establish the convergence for solving a broader family of non- compositional optimization problems.
  arXiv  Detail & Related papers  (2021-12-07T02:47:58Z)
• Understanding the Generalization of Adam in Learning Neural Networks with Proper Regularization [118.50301177912381]
  We show that Adam can converge to different solutions of the objective with provably different errors, even with weight decay globalization. We show that if convex, and the weight decay regularization is employed, any optimization algorithms including Adam will converge to the same solution.
  arXiv  Detail & Related papers  (2021-08-25T17:58:21Z)
• Adam$^+$: A Stochastic Method with Adaptive Variance Reduction [56.051001950733315]
  Adam is a widely used optimization method for deep learning applications. We propose a new method named Adam$+$ (pronounced as Adam-plus) Our empirical studies on various deep learning tasks, including image classification, language modeling, and automatic speech recognition, demonstrate that Adam$+$ significantly outperforms Adam.
  arXiv  Detail & Related papers  (2020-11-24T09:28:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.