Federated Learning Optimization: A Comparative Study of Data and Model Exchange Strategies in Dynamic Networks

• URL: http://arxiv.org/abs/2406.10798v1
• Date: Sun, 16 Jun 2024 03:46:23 GMT
• Title: Federated Learning Optimization: A Comparative Study of Data and Model Exchange Strategies in Dynamic Networks
• Authors: Alka Luqman, Yeow Wei Liang Brandon, Anupam Chattopadhyay,
• Abstract summary: We study the choices of exchanging raw data, synthetic data, or (partial) model updates among devices. Across various scenarios that we considered, time-limited knowledge transfer efficiency can differ by up to 9.08%.
• Score: 3.4179091429029382
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The promise and proliferation of large-scale dynamic federated learning gives rise to a prominent open question - is it prudent to share data or model across nodes, if efficiency of transmission and fast knowledge transfer are the prime objectives. This work investigates exactly that. Specifically, we study the choices of exchanging raw data, synthetic data, or (partial) model updates among devices. The implications of these strategies in the context of foundational models are also examined in detail. Accordingly, we obtain key insights about optimal data and model exchange mechanisms considering various environments with different data distributions and dynamic device and network connections. Across various scenarios that we considered, time-limited knowledge transfer efficiency can differ by up to 9.08\%, thus highlighting the importance of this work.

Related papers

• On Learnable Parameters of Optimal and Suboptimal Deep Learning Models [2.889799048595314]
  We study the structural and operational aspects of deep learning models. Our research focuses on the nuances of learnable parameters (weight) statistics, distribution, node interaction, and visualization.
  arXiv  Detail & Related papers  (2024-08-21T15:50:37Z)
• Leveraging Variation Theory in Counterfactual Data Augmentation for Optimized Active Learning [19.962212551963383]
  Active Learning (AL) allows models to learn interactively from user feedback. This paper introduces a counterfactual data augmentation approach to AL.
  arXiv  Detail & Related papers  (2024-08-07T14:55:04Z)
• MergeNet: Knowledge Migration across Heterogeneous Models, Tasks, and Modalities [72.68829963458408]
  We present MergeNet, which learns to bridge the gap of parameter spaces of heterogeneous models. The core mechanism of MergeNet lies in the parameter adapter, which operates by querying the source model's low-rank parameters. MergeNet is learned alongside both models, allowing our framework to dynamically transfer and adapt knowledge relevant to the current stage.
  arXiv  Detail & Related papers  (2024-04-20T08:34:39Z)
• FLASH: Federated Learning Across Simultaneous Heterogeneities [54.80435317208111]
  FLASH(Federated Learning Across Simultaneous Heterogeneities) is a lightweight and flexible client selection algorithm. It outperforms state-of-the-art FL frameworks under extensive sources of Heterogeneities. It achieves substantial and consistent improvements over state-of-the-art baselines.
  arXiv  Detail & Related papers  (2024-02-13T20:04:39Z)
• Personalized Federated Learning with Contextual Modulation and Meta-Learning [2.7716102039510564]
  Federated learning has emerged as a promising approach for training machine learning models on decentralized data sources. We propose a novel framework that combines federated learning with meta-learning techniques to enhance both efficiency and generalization capabilities.
  arXiv  Detail & Related papers  (2023-12-23T08:18:22Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• An Empirical Investigation of Commonsense Self-Supervision with Knowledge Graphs [67.23285413610243]
  Self-supervision based on the information extracted from large knowledge graphs has been shown to improve the generalization of language models. We study the effect of knowledge sampling strategies and sizes that can be used to generate synthetic data for adapting language models.
  arXiv  Detail & Related papers  (2022-05-21T19:49:04Z)
• DST: Dynamic Substitute Training for Data-free Black-box Attack [79.61601742693713]
  We propose a novel dynamic substitute training attack method to encourage substitute model to learn better and faster from the target model. We introduce a task-driven graph-based structure information learning constrain to improve the quality of generated training data.
  arXiv  Detail & Related papers  (2022-04-03T02:29:11Z)
• Weak Augmentation Guided Relational Self-Supervised Learning [80.0680103295137]
  We introduce a novel relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Our proposed method employs sharpened distribution of pairwise similarities among different instances as textitrelation metric. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures.
  arXiv  Detail & Related papers  (2022-03-16T16:14:19Z)
• Towards Federated Bayesian Network Structure Learning with Continuous Optimization [14.779035801521717]
  We present a cross-silo federated learning approach to estimate the structure of Bayesian network. We develop a distributed structure learning method based on continuous optimization.
  arXiv  Detail & Related papers  (2021-10-18T14:36:05Z)
• Probing transfer learning with a model of synthetic correlated datasets [11.53207294639557]
  Transfer learning can significantly improve the sample efficiency of neural networks. We re-think a solvable model of synthetic data as a framework for modeling correlation between data-sets. We show that our model can capture a range of salient features of transfer learning with real data.
  arXiv  Detail & Related papers  (2021-06-09T22:15:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.