The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset

• URL: http://arxiv.org/abs/2108.03508v1
• Date: Sat, 7 Aug 2021 19:37:43 GMT
• Title: The Effect of Training Parameters and Mechanisms on Decentralized Federated Learning based on MNIST Dataset
• Authors: Zhuofan Zhang, Mi Zhou, Kaicheng Niu, and Chaouki Abdallah
• Abstract summary: We introduce the notion of Decentralized Federated Learning (DFL) All experiments are run on the MNIST handwritten digits dataset. We observe failures in training when the variance between model weights is too large.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning is an algorithm suited for training models on decentralized data, but the requirement of a central "server" node is a bottleneck. In this document, we first introduce the notion of Decentralized Federated Learning (DFL). We then perform various experiments on different setups, such as changing model aggregation frequency, switching from independent and identically distributed (IID) dataset partitioning to non-IID partitioning with partial global sharing, using different optimization methods across clients, and breaking models into segments with partial sharing. All experiments are run on the MNIST handwritten digits dataset. We observe that those altered training procedures are generally robust, albeit non-optimal. We also observe failures in training when the variance between model weights is too large. The open-source experiment code is accessible through GitHub\footnote{Code was uploaded at \url{https://github.com/zhzhang2018/DecentralizedFL}}.

Related papers

• Exploiting Label Skews in Federated Learning with Model Concatenation [39.38427550571378]
  Federated Learning (FL) has emerged as a promising solution to perform deep learning on different data owners without exchanging raw data. Among different non-IID types, label skews have been challenging and common in image classification and other tasks. We propose FedConcat, a simple and effective approach that degrades these local models as the base of the global model.
  arXiv  Detail & Related papers  (2023-12-11T10:44:52Z)
• Towards More Suitable Personalization in Federated Learning via Decentralized Partial Model Training [67.67045085186797]
  Almost all existing systems have to face large communication burdens if the central FL server fails. It personalizes the "right" in the deep models by alternately updating the shared and personal parameters. To further promote the shared parameters aggregation process, we propose DFed integrating the local Sharpness Miniization.
  arXiv  Detail & Related papers  (2023-05-24T13:52:18Z)
• FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data [69.0785021613868]
  Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos. We propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages the gradient covariance and the geometric mean of Hessians to capture both inter-silo and intra-silo consistencies. This is relevant to various fields such as medical healthcare, computer vision, and the Internet of Things (IoT)
  arXiv  Detail & Related papers  (2022-05-19T03:32:03Z)
• Scatterbrained: A flexible and expandable pattern for decentralized machine learning [1.2891210250935146]
  Federated machine learning is a technique for training a model across multiple devices without exchanging data between them. We suggest a flexible framework for decentralizing the federated learning pattern, and provide an open-source, reference implementation compatible with PyTorch.
  arXiv  Detail & Related papers  (2021-12-14T19:39:35Z)
• RelaySum for Decentralized Deep Learning on Heterogeneous Data [71.36228931225362]
  In decentralized machine learning, workers compute model updates on their local data. Because the workers only communicate with few neighbors without central coordination, these updates propagate progressively over the network. This paradigm enables distributed training on networks without all-to-all connectivity, helping to protect data privacy as well as to reduce the communication cost of distributed training in data centers.
  arXiv  Detail & Related papers  (2021-10-08T14:55:32Z)
• Multi-Center Federated Learning [62.32725938999433]
  Federated learning (FL) can protect data privacy in distributed learning. It merely collects local gradients from users without access to their data. We propose a novel multi-center aggregation mechanism.
  arXiv  Detail & Related papers  (2021-08-19T12:20:31Z)
• Decentralized federated learning of deep neural networks on non-iid data [0.6335848702857039]
  We tackle the non-problem of learning a personalized deep learning model in a decentralized setting. We propose a method named Performance-Based Neighbor Selection (PENS) where clients with similar data detect each other and cooperate. PENS is able to achieve higher accuracies as compared to strong baselines.
  arXiv  Detail & Related papers  (2021-07-18T19:05:44Z)
• Decentralized Federated Learning via Mutual Knowledge Transfer [37.5341683644709]
  Decentralized federated learning (DFL) is a problem in the Internet of things (IoT) systems. We propose a mutual knowledge transfer (Def-KT) algorithm where local clients fuse models by transferring their learnt knowledge to each other. Our experiments on the MNIST, Fashion-MNIST, and CIFAR10 datasets reveal datasets that the proposed Def-KT algorithm significantly outperforms the baseline DFL methods.
  arXiv  Detail & Related papers  (2020-12-24T01:43:53Z)
• A decentralized aggregation mechanism for training deep learning models using smart contract system for bank loan prediction [0.1933681537640272]
  We present a solution to benefit from a distributed data setup in the case of training deep learning architectures by making use of a smart contract system. We propose a mechanism that aggregates together the intermediate representations obtained from local ANN models over a blockchain. The obtained performance, which is better than that of individual nodes, is at par with that of a centralized data setup.
  arXiv  Detail & Related papers  (2020-11-22T10:47:45Z)
• Ensemble Distillation for Robust Model Fusion in Federated Learning [72.61259487233214]
  Federated Learning (FL) is a machine learning setting where many devices collaboratively train a machine learning model. In most of the current training schemes the central model is refined by averaging the parameters of the server model and the updated parameters from the client side. We propose ensemble distillation for model fusion, i.e. training the central classifier through unlabeled data on the outputs of the models from the clients.
  arXiv  Detail & Related papers  (2020-06-12T14:49:47Z)
• Multi-Center Federated Learning [62.57229809407692]
  This paper proposes a novel multi-center aggregation mechanism for federated learning. It learns multiple global models from the non-IID user data and simultaneously derives the optimal matching between users and centers. Our experimental results on benchmark datasets show that our method outperforms several popular federated learning methods.
  arXiv  Detail & Related papers  (2020-05-03T09:14:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.