Homogeneous Learning: Self-Attention Decentralized Deep Learning

• URL: http://arxiv.org/abs/2110.05290v1
• Date: Mon, 11 Oct 2021 14:05:29 GMT
• Title: Homogeneous Learning: Self-Attention Decentralized Deep Learning
• Authors: Yuwei Sun and Hideya Ochiai
• Abstract summary: We propose a decentralized learning model called Homogeneous Learning (HL) for tackling non-IID data with a self-attention mechanism. HL can produce a better performance compared with standalone learning and greatly reduce both the total training rounds by 50.8% and the communication cost by 74.6%.
• Score: 0.6091702876917281
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Federated learning (FL) has been facilitating privacy-preserving deep learning in many walks of life such as medical image classification, network intrusion detection, and so forth. Whereas it necessitates a central parameter server for model aggregation, which brings about delayed model communication and vulnerability to adversarial attacks. A fully decentralized architecture like Swarm Learning allows peer-to-peer communication among distributed nodes, without the central server. One of the most challenging issues in decentralized deep learning is that data owned by each node are usually non-independent and identically distributed (non-IID), causing time-consuming convergence of model training. To this end, we propose a decentralized learning model called Homogeneous Learning (HL) for tackling non-IID data with a self-attention mechanism. In HL, training performs on each round's selected node, and the trained model of a node is sent to the next selected node at the end of each round. Notably, for the selection, the self-attention mechanism leverages reinforcement learning to observe a node's inner state and its surrounding environment's state, and find out which node should be selected to optimize the training. We evaluate our method with various scenarios for an image classification task. The result suggests that HL can produce a better performance compared with standalone learning and greatly reduce both the total training rounds by 50.8% and the communication cost by 74.6% compared with random policy-based decentralized learning for training on non-IID data.

Related papers

• Impact of network topology on the performance of Decentralized Federated Learning [4.618221836001186]
  Decentralized machine learning is gaining momentum, addressing infrastructure challenges and privacy concerns. This study investigates the interplay between network structure and learning performance using three network topologies and six data distribution methods. We highlight the challenges in transferring knowledge from peripheral to central nodes, attributed to a dilution effect during model aggregation.
  arXiv  Detail & Related papers  (2024-02-28T11:13:53Z)
• Communication-Efficient Decentralized Federated Learning via One-Bit Compressive Sensing [52.402550431781805]
  Decentralized federated learning (DFL) has gained popularity due to its practicality across various applications. Compared to the centralized version, training a shared model among a large number of nodes in DFL is more challenging. We develop a novel algorithm based on the framework of the inexact alternating direction method (iADM)
  arXiv  Detail & Related papers  (2023-08-31T12:22:40Z)
• Does Decentralized Learning with Non-IID Unlabeled Data Benefit from Self Supervision? [51.00034621304361]
  We study decentralized learning with unlabeled data through the lens of self-supervised learning (SSL) We study the effectiveness of contrastive learning algorithms under decentralized learning settings.
  arXiv  Detail & Related papers  (2022-10-20T01:32:41Z)
• DisPFL: Towards Communication-Efficient Personalized Federated Learning via Decentralized Sparse Training [84.81043932706375]
  We propose a novel personalized federated learning framework in a decentralized (peer-to-peer) communication protocol named Dis-PFL. Dis-PFL employs personalized sparse masks to customize sparse local models on the edge. We demonstrate that our method can easily adapt to heterogeneous local clients with varying computation complexities.
  arXiv  Detail & Related papers  (2022-06-01T02:20:57Z)
• Wireless Ad Hoc Federated Learning: A Fully Distributed Cooperative Machine Learning [2.724649101295584]
  Federated learning has allowed training of a global model by aggregating local models trained on local nodes. We propose a wireless ad hoc federated learning (WAFL) -- a fully distributed cooperative machine learning system. WAFL allowed the convergence of model parameters among the nodes toward generalization, even with opportunistic node contact scenarios.
  arXiv  Detail & Related papers  (2022-05-24T04:37:11Z)
• FedDKD: Federated Learning with Decentralized Knowledge Distillation [3.9084449541022055]
  We propose a novel framework of federated learning equipped with the process of decentralized knowledge distillation (FedDKD) We show that FedDKD outperforms the state-of-the-art methods with more efficient communication and training in a few DKD steps.
  arXiv  Detail & Related papers  (2022-05-02T07:54:07Z)
• RelaySum for Decentralized Deep Learning on Heterogeneous Data [71.36228931225362]
  In decentralized machine learning, workers compute model updates on their local data. Because the workers only communicate with few neighbors without central coordination, these updates propagate progressively over the network. This paradigm enables distributed training on networks without all-to-all connectivity, helping to protect data privacy as well as to reduce the communication cost of distributed training in data centers.
  arXiv  Detail & Related papers  (2021-10-08T14:55:32Z)
• Decentralized federated learning of deep neural networks on non-iid data [0.6335848702857039]
  We tackle the non-problem of learning a personalized deep learning model in a decentralized setting. We propose a method named Performance-Based Neighbor Selection (PENS) where clients with similar data detect each other and cooperate. PENS is able to achieve higher accuracies as compared to strong baselines.
  arXiv  Detail & Related papers  (2021-07-18T19:05:44Z)
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data [77.88594632644347]
  Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge. We propose a novel momentum-based method to mitigate this decentralized training difficulty.
  arXiv  Detail & Related papers  (2021-02-09T11:27:14Z)
• Decentralized Federated Learning Preserves Model and Data Privacy [77.454688257702]
  We propose a fully decentralized approach, which allows to share knowledge between trained models. Students are trained on the output of their teachers via synthetically generated input data. The results show that an untrained student model, trained on the teachers output reaches comparable F1-scores as the teacher.
  arXiv  Detail & Related papers  (2021-02-01T14:38:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.