When Computing Power Network Meets Distributed Machine Learning: An
Efficient Federated Split Learning Framework
- URL: http://arxiv.org/abs/2305.12979v1
- Date: Mon, 22 May 2023 12:36:52 GMT
- Title: When Computing Power Network Meets Distributed Machine Learning: An
Efficient Federated Split Learning Framework
- Authors: Xinjing Yuan, Lingjun Pu, Lei Jiao, Xiaofei Wang, Meijuan Yang,
Jingdong Xu
- Abstract summary: CPN-FedSL is a Federated Split Learning (FedSL) framework over Computing Power Network (CPN)
We build a dedicated model to capture the basic settings and learning characteristics (e.g., latency, flow, convergence)
- Score: 6.871107511111629
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: In this paper, we advocate CPN-FedSL, a novel and flexible Federated Split
Learning (FedSL) framework over Computing Power Network (CPN). We build a
dedicated model to capture the basic settings and learning characteristics
(e.g., training flow, latency and convergence). Based on this model, we
introduce Resource Usage Effectiveness (RUE), a novel performance metric
integrating training utility with system cost, and formulate a multivariate
scheduling problem that maxi?mizes RUE by comprehensively taking client
admission, model partition, server selection, routing and bandwidth allocation
into account (i.e., mixed-integer fractional programming). We design Refinery,
an efficient approach that first linearizes the fractional objective and
non-convex constraints, and then solves the transformed problem via a greedy
based rounding algorithm in multiple iterations. Extensive evaluations
corroborate that CPN-FedSL is superior to the standard and state-of-the-art
learning frameworks (e.g., FedAvg and SplitFed), and besides Refinery is
lightweight and significantly outperforms its variants and de facto heuristic
methods under a variety of settings.
Related papers
- Optimizing Value of Learning in Task-Oriented Federated Meta-Learning Systems [10.332182237773818]
A novel metric, metric value of learning (VoL) is introduced to assess the individual training needs across devices.
A task-level weight (TLW) is defined based on task-level considerations and fairness of FML training.
arXiv Detail & Related papers (2025-01-07T00:30:31Z) - Over-the-Air Fair Federated Learning via Multi-Objective Optimization [52.295563400314094]
We propose an over-the-air fair federated learning algorithm (OTA-FFL) to train fair FL models.
Experiments demonstrate the superiority of OTA-FFL in achieving fairness and robust performance.
arXiv Detail & Related papers (2025-01-06T21:16:51Z) - Federated Split Learning with Model Pruning and Gradient Quantization in Wireless Networks [7.439160287320074]
Federated split learning (FedSL) implements collaborative training across the edge devices and the server through model splitting.
We propose a lightweight FedSL scheme, that further alleviates the training burden on resource-constrained edge devices.
We conduct theoretical analysis to quantify the convergence performance of the proposed scheme.
arXiv Detail & Related papers (2024-12-09T11:43:03Z) - Read-ME: Refactorizing LLMs as Router-Decoupled Mixture of Experts with System Co-Design [59.00758127310582]
We propose a novel framework Read-ME that transforms pre-trained dense LLMs into smaller MoE models.
Our approach employs activation sparsity to extract experts.
Read-ME outperforms other popular open-source dense models of similar scales.
arXiv Detail & Related papers (2024-10-24T19:48:51Z) - ESFL: Efficient Split Federated Learning over Resource-Constrained Heterogeneous Wireless Devices [22.664980594996155]
Federated learning (FL) allows multiple parties (distributed devices) to train a machine learning model without sharing raw data.
We propose an efficient split federated learning algorithm (ESFL) to take full advantage of the powerful computing capabilities at a central server.
arXiv Detail & Related papers (2024-02-24T20:50:29Z) - Workflow Optimization for Parallel Split Learning [12.554265727169742]
Split learning (SL) has been proposed as a way to enable resource-constrained devices to train neural networks (NNs) and participate in federated learning (FL)
In parallel SL, multiple helpers can process model parts of one or more clients, thus, considerably reducing the maximum training time over all clients (makespan)
We propose a solution method based on the decomposition of the problem by leveraging its inherent symmetry, and a second one that is fully scalable.
arXiv Detail & Related papers (2024-02-01T14:16:10Z) - Unifying Synergies between Self-supervised Learning and Dynamic
Computation [53.66628188936682]
We present a novel perspective on the interplay between SSL and DC paradigms.
We show that it is feasible to simultaneously learn a dense and gated sub-network from scratch in a SSL setting.
The co-evolution during pre-training of both dense and gated encoder offers a good accuracy-efficiency trade-off.
arXiv Detail & Related papers (2023-01-22T17:12:58Z) - Predictive GAN-powered Multi-Objective Optimization for Hybrid Federated
Split Learning [56.125720497163684]
We propose a hybrid federated split learning framework in wireless networks.
We design a parallel computing scheme for model splitting without label sharing, and theoretically analyze the influence of the delayed gradient caused by the scheme on the convergence speed.
arXiv Detail & Related papers (2022-09-02T10:29:56Z) - Supernet Training for Federated Image Classification under System
Heterogeneity [15.2292571922932]
In this work, we propose a novel framework to consider both scenarios, namely Federation of Supernet Training (FedSup)
It is inspired by how averaging parameters in the model aggregation stage of Federated Learning (FL) is similar to weight-sharing in supernet training.
Under our framework, we present an efficient algorithm (E-FedSup) by sending the sub-model to clients in the broadcast stage for reducing communication costs and training overhead.
arXiv Detail & Related papers (2022-06-03T02:21:01Z) - An Expectation-Maximization Perspective on Federated Learning [75.67515842938299]
Federated learning describes the distributed training of models across multiple clients while keeping the data private on-device.
In this work, we view the server-orchestrated federated learning process as a hierarchical latent variable model where the server provides the parameters of a prior distribution over the client-specific model parameters.
We show that with simple Gaussian priors and a hard version of the well known Expectation-Maximization (EM) algorithm, learning in such a model corresponds to FedAvg, the most popular algorithm for the federated learning setting.
arXiv Detail & Related papers (2021-11-19T12:58:59Z) - Clustered Federated Learning via Generalized Total Variation
Minimization [83.26141667853057]
We study optimization methods to train local (or personalized) models for local datasets with a decentralized network structure.
Our main conceptual contribution is to formulate federated learning as total variation minimization (GTV)
Our main algorithmic contribution is a fully decentralized federated learning algorithm.
arXiv Detail & Related papers (2021-05-26T18:07:19Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.