Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings

• URL: http://arxiv.org/abs/2508.12327v1
• Date: Sun, 17 Aug 2025 10:54:01 GMT
• Title: Convergence Analysis of the Lion Optimizer in Centralized and Distributed Settings
• Abstract summary: We introduce the Lion with variance reduction, resulting in an enhanced convergence rate of $mathcalO(d1/2T-1/4)$.<n>We then analyze in distributed settings, where the standard and variance reduced version of the distributed Lion can obtain the convergence rates of $mathcalO(d1/2(nT)-1/4)$.
• Score: 16.68443164140878
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this paper, we analyze the convergence properties of the Lion optimizer. First, we establish that the Lion optimizer attains a convergence rate of $\mathcal{O}(d^{1/2}T^{-1/4})$ under standard assumptions, where $d$ denotes the problem dimension and $T$ is the iteration number. To further improve this rate, we introduce the Lion optimizer with variance reduction, resulting in an enhanced convergence rate of $\mathcal{O}(d^{1/2}T^{-1/3})$. We then analyze in distributed settings, where the standard and variance reduced version of the distributed Lion can obtain the convergence rates of $\mathcal{O}(d^{1/2}(nT)^{-1/4})$ and $\mathcal{O}(d^{1/2}(nT)^{-1/3})$, with $n$ denoting the number of nodes. Furthermore, we investigate a communication-efficient variant of the distributed Lion that ensures sign compression in both communication directions. By employing the unbiased sign operations, the proposed Lion variant and its variance reduction counterpart, achieve convergence rates of $\mathcal{O}\left( \max \left\{\frac{d^{1/4}}{T^{1/4}}, \frac{d^{1/10}}{n^{1/5}T^{1/5}} \right\} \right)$ and $\mathcal{O}\left( \frac{d^{1/4}}{T^{1/4}} \right)$, respectively.

Related papers

• CLion: Efficient Cautious Lion Optimizer with Enhanced Generalization [43.39364515909059]
  We show that Lion has a generalization error of $O(frac1NT)$.<n>We also show that our CLion has a fast convergence rate of $O(fracsqrtdddell)$.
  arXiv  Detail & Related papers  (2026-04-16T03:32:48Z)
• Decentralized Stochastic Nonconvex Optimization under the Relaxed Smoothness [21.090579632247707]
  We propose a novel algorithm called decentralized normalized gradient descent (DNS)<n>DNS can achieve an $bold$ilon point at each local agent.<n>We show the upper bounds on the sample complexity of $mathcal O(m-1(L_fsigma2Delta_fepsilon-4 + sigma2epsilon-2 + L_f-1)$ per agent.
  arXiv  Detail & Related papers  (2025-09-10T16:17:19Z)
• Improved Analysis for Sign-based Methods with Momentum Updates [15.679979249491087]
  We present enhanced analysis for sign-based optimization algorithms with momentum updates.<n>We show that a proposed momentum-based method yields convergence rates of $mathcalOleft( d1/2T-1/2 + dn-1/2 right)$ and $mathcalleft( max d1/4T-1/4 + dn-1/2 right)$.
  arXiv  Detail & Related papers  (2025-07-16T09:54:08Z)
• Proving the Limited Scalability of Centralized Distributed Optimization via a New Lower Bound Construction [57.93371273485736]
  We consider a centralized distributed learning setup where all workers jointly find an unbiased bound LDeltaepsilon2,$ better poly-logarithmically in $n$, even in the homogeneous (i.i.d.) case, where all workers access the same distribution.
  arXiv  Detail & Related papers  (2025-06-30T13:27:39Z)
• Convergence Rate Analysis of LION [54.28350823319057]
  LION converges iterations of $cal(sqrtdK-)$ measured by gradient Karush-Kuhn-T (sqrtdK-)$. We show that LION can achieve lower loss and higher performance compared to standard SGD.
  arXiv  Detail & Related papers  (2024-11-12T11:30:53Z)
• Efficient Sign-Based Optimization: Accelerating Convergence via Variance Reduction [16.82220229840038]
  We introduce two novel algorithms that attain improved convergence rates of $mathcalO(d1/2T-1/2 + dn-1/2)$ and $mathcalO(d1/4T-1/4)$ respectively.<n> Numerical experiments across different tasks validate the effectiveness of our proposed methods.
  arXiv  Detail & Related papers  (2024-06-01T16:38:43Z)
• Minimax Optimality of Score-based Diffusion Models: Beyond the Density Lower Bound Assumptions [11.222970035173372]
  kernel-based score estimator achieves an optimal mean square error of $widetildeOleft(n-1 t-fracd+22(tfracd2 vee 1)right) We show that a kernel-based score estimator achieves an optimal mean square error of $widetildeOleft(n-1/2 t-fracd4right)$ upper bound for the total variation error of the distribution of the sample generated by the diffusion model under a mere sub-Gaussian
  arXiv  Detail & Related papers  (2024-02-23T20:51:31Z)
• On the $O(\frac{\sqrt{d}}{T^{1/4}})$ Convergence Rate of RMSProp and Its Momentum Extension Measured by $\ell_1$ Norm [54.28350823319057]
  This paper considers the RMSProp and its momentum extension and establishes the convergence rate of $frac1Tsum_k=1T.<n>Our convergence rate matches the lower bound with respect to all the coefficients except the dimension $d$.<n>Our convergence rate can be considered to be analogous to the $frac1Tsum_k=1T.
  arXiv  Detail & Related papers  (2024-02-01T07:21:32Z)
• Polyak-Ruppert Averaged Q-Leaning is Statistically Efficient [90.14768299744792]
  We study synchronous Q-learning with Polyak-Ruppert averaging (a.k.a., averaged Q-leaning) in a $gamma$-discounted MDP. We establish normality for the iteration averaged $barboldsymbolQ_T$. In short, our theoretical analysis shows averaged Q-Leaning is statistically efficient.
  arXiv  Detail & Related papers  (2021-12-29T14:47:56Z)
• Optimal Mean Estimation without a Variance [103.26777953032537]
  We study the problem of heavy-tailed mean estimation in settings where the variance of the data-generating distribution does not exist. We design an estimator which attains the smallest possible confidence interval as a function of $n,d,delta$.
  arXiv  Detail & Related papers  (2020-11-24T22:39:21Z)
• Sparse sketches with small inversion bias [79.77110958547695]
  Inversion bias arises when averaging estimates of quantities that depend on the inverse covariance. We develop a framework for analyzing inversion bias, based on our proposed concept of an $(epsilon,delta)$-unbiased estimator for random matrices. We show that when the sketching matrix $S$ is dense and has i.i.d. sub-gaussian entries, the estimator $(epsilon,delta)$-unbiased for $(Atop A)-1$ with a sketch of size $m=O(d+sqrt d/
  arXiv  Detail & Related papers  (2020-11-21T01:33:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.