Impact of network topology on the performance of Decentralized Federated Learning

• URL: http://arxiv.org/abs/2402.18606v1
• Date: Wed, 28 Feb 2024 11:13:53 GMT
• Title: Impact of network topology on the performance of Decentralized Federated Learning
• Authors: Luigi Palmieri and Chiara Boldrini and Lorenzo Valerio and Andrea Passarella and Marco Conti
• Abstract summary: Decentralized machine learning is gaining momentum, addressing infrastructure challenges and privacy concerns. This study investigates the interplay between network structure and learning performance using three network topologies and six data distribution methods. We highlight the challenges in transferring knowledge from peripheral to central nodes, attributed to a dilution effect during model aggregation.
• Score: 4.618221836001186
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Fully decentralized learning is gaining momentum for training AI models at the Internet's edge, addressing infrastructure challenges and privacy concerns. In a decentralized machine learning system, data is distributed across multiple nodes, with each node training a local model based on its respective dataset. The local models are then shared and combined to form a global model capable of making accurate predictions on new data. Our exploration focuses on how different types of network structures influence the spreading of knowledge - the process by which nodes incorporate insights gained from learning patterns in data available on other nodes across the network. Specifically, this study investigates the intricate interplay between network structure and learning performance using three network topologies and six data distribution methods. These methods consider different vertex properties, including degree centrality, betweenness centrality, and clustering coefficient, along with whether nodes exhibit high or low values of these metrics. Our findings underscore the significance of global centrality metrics (degree, betweenness) in correlating with learning performance, while local clustering proves less predictive. We highlight the challenges in transferring knowledge from peripheral to central nodes, attributed to a dilution effect during model aggregation. Additionally, we observe that central nodes exert a pull effect, facilitating the spread of knowledge. In examining degree distribution, hubs in Barabasi-Albert networks positively impact learning for central nodes but exacerbate dilution when knowledge originates from peripheral nodes. Finally, we demonstrate the formidable challenge of knowledge circulation outside of segregated communities.

Related papers

• Robustness of Decentralised Learning to Nodes and Data Disruption [4.062458976723649]
  We study the effect of nodes' disruption on the collective learning process. Our results show that decentralised learning processes are remarkably robust to network disruption.
  arXiv  Detail & Related papers  (2024-05-03T12:14:48Z)
• Initialisation and Network Effects in Decentralised Federated Learning [1.5961625979922607]
  Decentralised federated learning enables collaborative training of individual machine learning models on a distributed network of communicating devices. This approach avoids central coordination, enhances data privacy and eliminates the risk of a single point of failure. We propose a strategy for uncoordinated initialisation of the artificial neural networks based on the distribution of eigenvector centralities of the underlying communication network.
  arXiv  Detail & Related papers  (2024-03-23T14:24:36Z)
• An effective theory of collective deep learning [1.3812010983144802]
  We introduce a minimal model that condenses several recent decentralized algorithms. We derive an effective theory for linear networks to show that the coarse-grained behavior of our system is equivalent to a deformed Ginzburg-Landau model. We validate the theory in coupled ensembles of realistic neural networks trained on the MNIST dataset.
  arXiv  Detail & Related papers  (2023-10-19T14:58:20Z)
• The effect of network topologies on fully decentralized learning: a preliminary investigation [2.9592782993171918]
  In a decentralized machine learning system, data is partitioned among multiple devices or nodes, each of which trains a local model using its own data. We investigate how different types of topologies impact the "spreading of knowledge" Specifically, we highlight the different roles in this process of more or less connected nodes (hubs and leaves)
  arXiv  Detail & Related papers  (2023-07-29T09:39:17Z)
• Distributed Learning over Networks with Graph-Attention-Based Personalization [49.90052709285814]
  We propose a graph-based personalized algorithm (GATTA) for distributed deep learning. In particular, the personalized model in each agent is composed of a global part and a node-specific part. By treating each agent as one node in a graph the node-specific parameters as its features, the benefits of the graph attention mechanism can be inherited.
  arXiv  Detail & Related papers  (2023-05-22T13:48:30Z)
• FedILC: Weighted Geometric Mean and Invariant Gradient Covariance for Federated Learning on Non-IID Data [69.0785021613868]
  Federated learning is a distributed machine learning approach which enables a shared server model to learn by aggregating the locally-computed parameter updates with the training data from spatially-distributed client silos. We propose the Federated Invariant Learning Consistency (FedILC) approach, which leverages the gradient covariance and the geometric mean of Hessians to capture both inter-silo and intra-silo consistencies. This is relevant to various fields such as medical healthcare, computer vision, and the Internet of Things (IoT)
  arXiv  Detail & Related papers  (2022-05-19T03:32:03Z)
• FedDKD: Federated Learning with Decentralized Knowledge Distillation [3.9084449541022055]
  We propose a novel framework of federated learning equipped with the process of decentralized knowledge distillation (FedDKD) We show that FedDKD outperforms the state-of-the-art methods with more efficient communication and training in a few DKD steps.
  arXiv  Detail & Related papers  (2022-05-02T07:54:07Z)
• Anomaly Detection on Attributed Networks via Contrastive Self-Supervised Learning [50.24174211654775]
  We present a novel contrastive self-supervised learning framework for anomaly detection on attributed networks. Our framework fully exploits the local information from network data by sampling a novel type of contrastive instance pair. A graph neural network-based contrastive learning model is proposed to learn informative embedding from high-dimensional attributes and local structure.
  arXiv  Detail & Related papers  (2021-02-27T03:17:20Z)
• Consensus Control for Decentralized Deep Learning [72.50487751271069]
  Decentralized training of deep learning models enables on-device learning over networks, as well as efficient scaling to large compute clusters. We show in theory that when the training consensus distance is lower than a critical quantity, decentralized training converges as fast as the centralized counterpart. Our empirical insights allow the principled design of better decentralized training schemes that mitigate the performance drop.
  arXiv  Detail & Related papers  (2021-02-09T13:58:33Z)
• Quasi-Global Momentum: Accelerating Decentralized Deep Learning on Heterogeneous Data [77.88594632644347]
  Decentralized training of deep learning models is a key element for enabling data privacy and on-device learning over networks. In realistic learning scenarios, the presence of heterogeneity across different clients' local datasets poses an optimization challenge. We propose a novel momentum-based method to mitigate this decentralized training difficulty.
  arXiv  Detail & Related papers  (2021-02-09T11:27:14Z)
• Learning Connectivity of Neural Networks from a Topological Perspective [80.35103711638548]
  We propose a topological perspective to represent a network into a complete graph for analysis. By assigning learnable parameters to the edges which reflect the magnitude of connections, the learning process can be performed in a differentiable manner. This learning process is compatible with existing networks and owns adaptability to larger search spaces and different tasks.
  arXiv  Detail & Related papers  (2020-08-19T04:53:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.