Communication-Efficient Gluon in Federated Learning

• URL: http://arxiv.org/abs/2604.10689v1
• Date: Sun, 12 Apr 2026 15:30:28 GMT
• Title: Communication-Efficient Gluon in Federated Learning
• Abstract summary: We investigate Gluon, which is an extension of Muon under the more general layer-wise $(L0, L1)$-smooth setting.<n>In order to reduce the compression error, we employ the variance reduced technique in SARAH in our compressed methods.
• Score: 89.94131981227609
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent developments have shown that Muon-type optimizers based on linear minimization oracles (LMOs) over non-Euclidean norm balls have the potential to get superior practical performance than Adam-type methods in the training of large language models. Since large-scale neural networks are trained across massive machines, communication cost becomes the bottleneck. To address this bottleneck, we investigate Gluon, which is an extension of Muon under the more general layer-wise $(L^0, L^1)$-smooth setting, with both unbiased and contraction compressors. In order to reduce the compression error, we employ the variance reduced technique in SARAH in our compressed methods. The convergence rates and improved communication cost are achieved under certain conditions. As a byproduct, a new variance reduced algorithm with faster convergence rate than Gluon is obtained. We also incorporate momentum variance reduction (MVR) to these compressed algorithms and comparable communication cost is derived under weaker conditions when $L_i^1 \neq 0$. Finally, several numerical experiments are conducted to verify the superior performance of our compressed algorithms in terms of communication cost.

Related papers

• Performance of Neural and Polynomial Operator Surrogates [0.822469542459168]
  Polynomial surrogates achieve substantially better data efficiency for smooth input fields.<n>For rough inputs, the Fourier neural operator displays the fastest convergence rates.<n> ative-informed training consistently improves data efficiency over standard $L2_$ training.
  arXiv  Detail & Related papers  (2026-04-01T09:46:25Z)
• Muon is Provably Faster with Momentum Variance Reduction [55.388203260208485]
  Recent empirical research has demonstrated that deep learnings based on the linear linear oracle (LMO) over specifically chosen Non-Eudean.<n>Adam-type training methods outperform the minimization of large language models.
  arXiv  Detail & Related papers  (2025-12-18T14:38:39Z)
• Error Feedback for Muon and Friends [80.90330715662961]
  We introduce EF21-Muon, the first communication-efficient, non-Euclidean LMO-based with rigorous convergence guarantees.<n>Our theory covers non-Euclidean smooth and the more general $(L0, L1)$-smooth setting, matching best-known Euclidean rates and enabling faster convergence under suitable norm choices.
  arXiv  Detail & Related papers  (2025-10-01T08:20:08Z)
• Compressed and Sparse Models for Non-Convex Decentralized Learning [6.14375469212514]
  Frequent model communication is a significant bottleneck to the efficiency of decentralized machine learning. We present Malcom-PSGD, a novel decentralized ML algorithm that combines model sparsity with a gradient gradient. Our method reduces communication costs by approximately $75%$ when compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-11-09T21:55:53Z)
• Communication-Efficient Adam-Type Algorithms for Distributed Data Mining [93.50424502011626]
  We propose a class of novel distributed Adam-type algorithms (emphi.e., SketchedAMSGrad) utilizing sketching. Our new algorithm achieves a fast convergence rate of $O(frac1sqrtnT + frac1(k/d)2 T)$ with the communication cost of $O(k log(d))$ at each iteration.
  arXiv  Detail & Related papers  (2022-10-14T01:42:05Z)
• BEER: Fast $O(1/T)$ Rate for Decentralized Nonconvex Optimization with Communication Compression [37.20712215269538]
  Communication efficiency has been widely recognized as the bottleneck for large-scale decentralized machine learning applications. This paper proposes BEER, which adopts communication with gradient tracking, shows it converges at a faster rate.
  arXiv  Detail & Related papers  (2022-01-31T16:14:09Z)
• Permutation Compressors for Provably Faster Distributed Nonconvex Optimization [68.8204255655161]
  We show that the MARINA method of Gorbunov et al (2021) can be considered as a state-of-the-art method in terms of theoretical communication complexity. Theory of MARINA to support the theory of potentially em correlated compressors, extends to the method beyond the classical independent compressors setting.
  arXiv  Detail & Related papers  (2021-10-07T09:38:15Z)
• Rethinking gradient sparsification as total error minimization [0.0]
  Gradient compression is a widely-established remedy to tackle the communication bottleneck in distributed training training networks (DNNs) We argue that the benefits of gradient sparsification, especially for DNNs, are necessary -- one that moves from perit optimality to consider optimality for the entire training.
  arXiv  Detail & Related papers  (2021-08-02T14:52:42Z)
• A Linearly Convergent Algorithm for Decentralized Optimization: Sending Less Bits for Free! [72.31332210635524]
  Decentralized optimization methods enable on-device training of machine learning models without a central coordinator. We propose a new randomized first-order method which tackles the communication bottleneck by applying randomized compression operators. We prove that our method can solve the problems without any increase in the number of communications compared to the baseline.
  arXiv  Detail & Related papers  (2020-11-03T13:35:53Z)
• Communication-Efficient Distributed Stochastic AUC Maximization with Deep Neural Networks [50.42141893913188]
  We study a distributed variable for large-scale AUC for a neural network as with a deep neural network. Our model requires a much less number of communication rounds and still a number of communication rounds in theory. Our experiments on several datasets show the effectiveness of our theory and also confirm our theory.
  arXiv  Detail & Related papers  (2020-05-05T18:08:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.