Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning

• URL: http://arxiv.org/abs/2412.03752v1
• Date: Wed, 04 Dec 2024 22:46:06 GMT
• Title: Beyond Local Sharpness: Communication-Efficient Global Sharpness-aware Minimization for Federated Learning
• Authors: Debora Caldarola, Pietro Cagnasso, Barbara Caputo, Marco Ciccone,
• Abstract summary: Federated learning (FL) enables collaborative model training with privacy preservation. Recent approaches use client-side sharpness-aware minimization (SAM) to encourage flatter minima. This work introduces FedGloSS, a novel FL approach that prioritizes the optimization of global sharpness on the server.
• Score: 23.733942517577134
• License:
• Abstract: Federated learning (FL) enables collaborative model training with privacy preservation. Data heterogeneity across edge devices (clients) can cause models to converge to sharp minima, negatively impacting generalization and robustness. Recent approaches use client-side sharpness-aware minimization (SAM) to encourage flatter minima, but the discrepancy between local and global loss landscapes often undermines their effectiveness, as optimizing for local sharpness does not ensure global flatness. This work introduces FedGloSS (Federated Global Server-side Sharpness), a novel FL approach that prioritizes the optimization of global sharpness on the server, using SAM. To reduce communication overhead, FedGloSS cleverly approximates sharpness using the previous global gradient, eliminating the need for additional client communication. Our extensive evaluations demonstrate that FedGloSS consistently reaches flatter minima and better performance compared to state-of-the-art FL methods across various federated vision benchmarks.

Related papers

• Locally Estimated Global Perturbations are Better than Local Perturbations for Federated Sharpness-aware Minimization [81.32266996009575]
  In federated learning (FL), the multi-step update and data heterogeneity among clients often lead to a loss landscape with sharper minima. We propose FedLESAM, a novel algorithm that locally estimates the direction of global perturbation on client side.
  arXiv  Detail & Related papers  (2024-05-29T08:46:21Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)
• Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape [59.841889495864386]
  In federated learning (FL), a cluster of local clients are chaired under the coordination of a global server. Clients are prone to overfit into their own optima, which extremely deviates from the global objective. ttfamily FedSMOO adopts a dynamic regularizer to guarantee the local optima towards the global objective. Our theoretical analysis indicates that ttfamily FedSMOO achieves fast $mathcalO (1/T)$ convergence rate with low bound generalization.
  arXiv  Detail & Related papers  (2023-05-19T10:47:44Z)
• FedLAP-DP: Federated Learning by Sharing Differentially Private Loss Approximations [53.268801169075836]
  We propose FedLAP-DP, a novel privacy-preserving approach for federated learning. A formal privacy analysis demonstrates that FedLAP-DP incurs the same privacy costs as typical gradient-sharing schemes. Our approach presents a faster convergence speed compared to typical gradient-sharing methods.
  arXiv  Detail & Related papers  (2023-02-02T12:56:46Z)
• Generalized Federated Learning via Sharpness Aware Minimization [22.294290071999736]
  We propose a general, effective algorithm, textttFedSAM, based on Sharpness Aware Minimization (SAM) local, and develop a momentum FL algorithm to bridge local and global models. Empirically, our proposed algorithms substantially outperform existing FL studies and significantly decrease the learning deviation.
  arXiv  Detail & Related papers  (2022-06-06T13:54:41Z)
• Sharpness-Aware Training for Free [163.1248341911413]
  SharpnessAware Minimization (SAM) has shown that minimizing a sharpness measure, which reflects the geometry of the loss landscape, can significantly reduce the generalization error. Sharpness-Aware Training Free (SAF) mitigates the sharp landscape at almost zero computational cost over the base. SAF ensures the convergence to a flat minimum with improved capabilities.
  arXiv  Detail & Related papers  (2022-05-27T16:32:43Z)
• FedCos: A Scene-adaptive Federated Optimization Enhancement for Performance Improvement [11.687451505965655]
  We propose FedCos, which reduces the directional inconsistency of local models by introducing a cosine-similarity penalty. We show that FedCos outperforms the well-known baselines and can enhance them under a variety of FL scenes. With the help of FedCos, multiple FL methods require significantly fewer communication rounds than before to obtain a model with comparable performance.
  arXiv  Detail & Related papers  (2022-04-07T02:59:54Z)
• Sparse Federated Learning with Hierarchical Personalized Models [24.763028713043468]
  Federated learning (FL) can achieve privacy-safe and reliable collaborative training without collecting users' private data. We propose a personalized FL algorithm using a hierarchical proximal mapping based on the moreau envelop, named sparse federated learning with hierarchical personalized models (sFedHP) A continuously differentiable approximated L1-norm is also used as the sparse constraint to reduce the communication cost.
  arXiv  Detail & Related papers  (2022-03-25T09:06:42Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.