Structured Cooperative Learning with Graphical Model Priors
- URL: http://arxiv.org/abs/2306.09595v2
- Date: Wed, 21 Jun 2023 14:43:27 GMT
- Title: Structured Cooperative Learning with Graphical Model Priors
- Authors: Shuangtong Li, Tianyi Zhou, Xinmei Tian, Dacheng Tao
- Abstract summary: We study how to train personalized models for different tasks on decentralized devices with limited local data.
We propose "Structured Cooperative Learning (SCooL)", in which a cooperation graph across devices is generated by a graphical model.
We evaluate SCooL and compare it with existing decentralized learning methods on an extensive set of benchmarks.
- Score: 98.53322192624594
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We study how to train personalized models for different tasks on
decentralized devices with limited local data. We propose "Structured
Cooperative Learning (SCooL)", in which a cooperation graph across devices is
generated by a graphical model prior to automatically coordinate mutual
learning between devices. By choosing graphical models enforcing different
structures, we can derive a rich class of existing and novel decentralized
learning algorithms via variational inference. In particular, we show three
instantiations of SCooL that adopt Dirac distribution, stochastic block model
(SBM), and attention as the prior generating cooperation graphs. These EM-type
algorithms alternate between updating the cooperation graph and cooperative
learning of local models. They can automatically capture the cross-task
correlations among devices by only monitoring their model updating in order to
optimize the cooperation graph. We evaluate SCooL and compare it with existing
decentralized learning methods on an extensive set of benchmarks, on which
SCooL always achieves the highest accuracy of personalized models and
significantly outperforms other baselines on communication efficiency. Our code
is available at https://github.com/ShuangtongLi/SCooL.
Related papers
- Self-Supervised Contrastive Graph Clustering Network via Structural Information Fusion [15.293684479404092]
We propose a novel deep graph clustering method called CGCN.
Our approach introduces contrastive signals and deep structural information into the pre-training process.
Our method has been experimentally validated on multiple real-world graph datasets.
arXiv Detail & Related papers (2024-08-08T09:49:26Z) - MAPL: Model Agnostic Peer-to-peer Learning [2.9221371172659616]
We introduce Model Agnostic Peer-to-peer Learning (MAPL) to simultaneously learn heterogeneous personalized models and a collaboration graph.
MAPL is comprised of two main modules: (i) local-level Personalized Model Learning (PML), leveraging a combination of intra- and inter-client contrastive losses; (ii) network-wide decentralized Collaborative Graph Learning (CGL) dynamically refining collaboration weights based on local task similarities.
arXiv Detail & Related papers (2024-03-28T19:17:54Z) - Vertical Federated Learning over Cloud-RAN: Convergence Analysis and
System Optimization [82.12796238714589]
We propose a novel cloud radio access network (Cloud-RAN) based vertical FL system to enable fast and accurate model aggregation.
We characterize the convergence behavior of the vertical FL algorithm considering both uplink and downlink transmissions.
We establish a system optimization framework by joint transceiver and fronthaul quantization design, for which successive convex approximation and alternate convex search based system optimization algorithms are developed.
arXiv Detail & Related papers (2023-05-04T09:26:03Z) - Unrolled Graph Learning for Multi-Agent Collaboration [37.239120967721156]
We propose a distributed multi-agent learning model inspired by human collaboration.
Agents can autonomously detect suitable collaborators and refer to collaborators' model for better performance.
arXiv Detail & Related papers (2022-10-31T07:05:44Z) - Decentralized Training of Foundation Models in Heterogeneous
Environments [77.47261769795992]
Training foundation models, such as GPT-3 and PaLM, can be extremely expensive.
We present the first study of training large foundation models with model parallelism in a decentralized regime over a heterogeneous network.
arXiv Detail & Related papers (2022-06-02T20:19:51Z) - Data-heterogeneity-aware Mixing for Decentralized Learning [63.83913592085953]
We characterize the dependence of convergence on the relationship between the mixing weights of the graph and the data heterogeneity across nodes.
We propose a metric that quantifies the ability of a graph to mix the current gradients.
Motivated by our analysis, we propose an approach that periodically and efficiently optimize the metric.
arXiv Detail & Related papers (2022-04-13T15:54:35Z) - Parallel Successive Learning for Dynamic Distributed Model Training over
Heterogeneous Wireless Networks [50.68446003616802]
Federated learning (FedL) has emerged as a popular technique for distributing model training over a set of wireless devices.
We develop parallel successive learning (PSL), which expands the FedL architecture along three dimensions.
Our analysis sheds light on the notion of cold vs. warmed up models, and model inertia in distributed machine learning.
arXiv Detail & Related papers (2022-02-07T05:11:01Z) - Towards Similarity-Aware Time-Series Classification [51.2400839966489]
We study time-series classification (TSC), a fundamental task of time-series data mining.
We propose Similarity-Aware Time-Series Classification (SimTSC), a framework that models similarity information with graph neural networks (GNNs)
arXiv Detail & Related papers (2022-01-05T02:14:57Z) - A Personalized Federated Learning Algorithm: an Application in Anomaly
Detection [0.6700873164609007]
Federated Learning (FL) has recently emerged as a promising method to overcome data privacy and transmission issues.
In FL, datasets collected from different devices or sensors are used to train local models (clients) each of which shares its learning with a centralized model (server)
This paper proposes a novel Personalized FedAvg (PC-FedAvg) which aims to control weights communication and aggregation augmented with a tailored learning algorithm to personalize the resulting models at each client.
arXiv Detail & Related papers (2021-11-04T04:57:11Z) - Privacy-Preserving Object Detection & Localization Using Distributed
Machine Learning: A Case Study of Infant Eyeblink Conditioning [1.3022864665437273]
We explore scalable distributed-training versions of two algorithms commonly used in object detection.
The application of both algorithms in the medical field is examined using a paradigm from the fields of psychology and neuroscience.
arXiv Detail & Related papers (2020-10-14T17:33:28Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.