Federated Learning with Manifold Regularization and Normalized Update Reaggregation

• URL: http://arxiv.org/abs/2311.05924v1
• Date: Fri, 10 Nov 2023 08:14:27 GMT
• Title: Federated Learning with Manifold Regularization and Normalized Update Reaggregation
• Authors: Xuming An, Li Shen, Han Hu, Yong Luo
• Abstract summary: Federated Learning (FL) is a collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. In FL, the model inconsistency caused by the local data across clients results in the near-orthogonality of client updates. We propose FedMRUR by adopting the emerging manifold model fusion scheme and a new global update to alleviate the negative impacts.
• Score: 22.885899072143676
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning (FL) is an emerging collaborative machine learning framework where multiple clients train the global model without sharing their own datasets. In FL, the model inconsistency caused by the local data heterogeneity across clients results in the near-orthogonality of client updates, which leads to the global update norm reduction and slows down the convergence. Most previous works focus on eliminating the difference of parameters (or gradients) between the local and global models, which may fail to reflect the model inconsistency due to the complex structure of the machine learning model and the Euclidean space's limitation in meaningful geometric representations. In this paper, we propose FedMRUR by adopting the manifold model fusion scheme and a new global optimizer to alleviate the negative impacts. Concretely, FedMRUR adopts a hyperbolic graph manifold regularizer enforcing the representations of the data in the local and global models are close to each other in a low-dimensional subspace. Because the machine learning model has the graph structure, the distance in hyperbolic space can reflect the model bias better than the Euclidean distance. In this way, FedMRUR exploits the manifold structures of the representations to significantly reduce the model inconsistency. FedMRUR also aggregates the client updates norms as the global update norm, which can appropriately enlarge each client's contribution to the global update, thereby mitigating the norm reduction introduced by the near-orthogonality of client updates. Furthermore, we theoretically prove that our algorithm can achieve a linear speedup property for non-convex setting under partial client participation.Experiments demonstrate that FedMRUR can achieve a new state-of-the-art (SOTA) accuracy with less communication.

Related papers

• An Aggregation-Free Federated Learning for Tackling Data Heterogeneity [50.44021981013037]
  Federated Learning (FL) relies on the effectiveness of utilizing knowledge from distributed datasets. Traditional FL methods adopt an aggregate-then-adapt framework, where clients update local models based on a global model aggregated by the server from the previous training round. We introduce FedAF, a novel aggregation-free FL algorithm.
  arXiv  Detail & Related papers  (2024-04-29T05:55:23Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Rethinking Client Drift in Federated Learning: A Logit Perspective [125.35844582366441]
  Federated Learning (FL) enables multiple clients to collaboratively learn in a distributed way, allowing for privacy protection. We find that the difference in logits between the local and global models increases as the model is continuously updated. We propose a new algorithm, named FedCSD, a Class prototype Similarity Distillation in a federated framework to align the local and global models.
  arXiv  Detail & Related papers  (2023-08-20T04:41:01Z)
• Federated Learning for Semantic Parsing: Task Formulation, Evaluation Setup, New Algorithms [29.636944156801327]
  Multiple clients collaboratively train one global model without sharing their semantic parsing data. Lorar adjusts each client's contribution to the global model update based on its training loss reduction during each round. Clients with smaller datasets enjoy larger performance gains.
  arXiv  Detail & Related papers  (2023-05-26T19:25:49Z)
• Dynamic Regularized Sharpness Aware Minimization in Federated Learning: Approaching Global Consistency and Smooth Landscape [59.841889495864386]
  In federated learning (FL), a cluster of local clients are chaired under the coordination of a global server. Clients are prone to overfit into their own optima, which extremely deviates from the global objective. ttfamily FedSMOO adopts a dynamic regularizer to guarantee the local optima towards the global objective. Our theoretical analysis indicates that ttfamily FedSMOO achieves fast $mathcalO (1/T)$ convergence rate with low bound generalization.
  arXiv  Detail & Related papers  (2023-05-19T10:47:44Z)
• Is Aggregation the Only Choice? Federated Learning via Layer-wise Model Recombination [33.12164201146458]
  We propose a novel and FL paradigm named FedMR (Federated Model Recombination) The goal of FedMR is to guide the recombined models to be trained towards a flat area. Compared with state-of-the-art FL methods, FedMR can significantly improve the inference accuracy without exposing privacy of each client.
  arXiv  Detail & Related papers  (2023-05-18T05:58:24Z)
• Understanding and Improving Model Averaging in Federated Learning on Heterogeneous Data [9.792805355704203]
  We study the loss landscape of model averaging in federated learning (FL) We decompose the expected loss of the global model into five factors related to the client models. We propose utilizing IMA on the global model at the late training phase to reduce its deviation from the expected speed.
  arXiv  Detail & Related papers  (2023-05-13T06:19:55Z)
• Towards Understanding and Mitigating Dimensional Collapse in Heterogeneous Federated Learning [112.69497636932955]
  Federated learning aims to train models across different clients without the sharing of data for privacy considerations. We study how data heterogeneity affects the representations of the globally aggregated models. We propose sc FedDecorr, a novel method that can effectively mitigate dimensional collapse in federated learning.
  arXiv  Detail & Related papers  (2022-10-01T09:04:17Z)
• Gradient Masked Averaging for Federated Learning [24.687254139644736]
  Federated learning allows a large number of clients with heterogeneous data to coordinate learning of a unified global model. Standard FL algorithms involve averaging of model parameters or gradient updates to approximate the global model at the server. We propose a gradient masked averaging approach for FL as an alternative to the standard averaging of client updates.
  arXiv  Detail & Related papers  (2022-01-28T08:42:43Z)
• Federated Submodel Optimization for Hot and Cold Data Features [41.99190452773989]
  We study practical data characteristics underlying federated learning, where non-i.i.d. data from clients have sparse features, and a certain client's local data normally involves only a small part of the full model. Due to data sparsity, the classical federated averaging (FedAvg) algorithm or its variants will be severely slowed down. We propose federated submodel averaging (FedSubAvg), ensuring that the expectation of the global update of each model parameter is equal to the average of the local updates of the clients.
  arXiv  Detail & Related papers  (2021-09-16T03:54:54Z)
• A Bayesian Federated Learning Framework with Online Laplace Approximation [144.7345013348257]
  Federated learning allows multiple clients to collaboratively learn a globally shared model. We propose a novel FL framework that uses online Laplace approximation to approximate posteriors on both the client and server side. We achieve state-of-the-art results on several benchmarks, clearly demonstrating the advantages of the proposed method.
  arXiv  Detail & Related papers  (2021-02-03T08:36:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.