CoBo: Collaborative Learning via Bilevel Optimization

• URL: http://arxiv.org/abs/2409.05539v1
• Date: Mon, 9 Sep 2024 11:59:42 GMT
• Title: CoBo: Collaborative Learning via Bilevel Optimization
• Authors: Diba Hashemi, Lie He, Martin Jaggi,
• Abstract summary: Collaborative learning is an important tool to train multiple clients more effectively by enabling communication among clients. In this paper, we model client-selection and model-training as two interconnected optimization problems. We introduce CoBo, a scalable and elastic, SGD-type alternating optimization algorithm that efficiently addresses these problem with theoretical convergence guarantees.
• Score: 36.8801794168583
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Collaborative learning is an important tool to train multiple clients more effectively by enabling communication among clients. Identifying helpful clients, however, presents challenging and often introduces significant overhead. In this paper, we model client-selection and model-training as two interconnected optimization problems, proposing a novel bilevel optimization problem for collaborative learning. We introduce CoBo, a scalable and elastic, SGD-type alternating optimization algorithm that efficiently addresses these problem with theoretical convergence guarantees. Empirically, CoBo achieves superior performance, surpassing popular personalization algorithms by 9.3% in accuracy on a task with high heterogeneity, involving datasets distributed among 80 clients.

Related papers

• Adaptive collaboration for online personalized distributed learning with heterogeneous clients [22.507916490976044]
  We study the problem of online personalized learning with $N$ statistically heterogeneous clients collaborating to accelerate local training.<n>An important challenge in this setting is to select relevant collaborators to reduce variance while mitigating the introduced bias.
  arXiv  Detail & Related papers  (2025-07-09T13:44:27Z)
• Federated Multimodal Learning with Dual Adapters and Selective Pruning for Communication and Computational Efficiency [6.0860246234554545]
  Federated Learning (FL) enables collaborative learning across distributed clients while preserving data privacy. We propose a novel framework designed to tackle these challenges by introducing a dual-adapter approach.
  arXiv  Detail & Related papers  (2025-03-10T17:21:33Z)
• Decentralized Personalized Federated Learning [4.5836393132815045]
  We focus on creating a collaboration graph that guides each client in selecting suitable collaborators for training personalized models. Unlike traditional methods, our formulation identifies collaborators at a granular level by considering greedy relations of clients. We achieve this through a bi-level optimization framework that employs a constrained algorithm.
  arXiv  Detail & Related papers  (2024-06-10T17:58:48Z)
• Emulating Full Client Participation: A Long-Term Client Selection Strategy for Federated Learning [48.94952630292219]
  We propose a novel client selection strategy designed to emulate the performance achieved with full client participation. In a single round, we select clients by minimizing the gradient-space estimation error between the client subset and the full client set. In multi-round selection, we introduce a novel individual fairness constraint, which ensures that clients with similar data distributions have similar frequencies of being selected.
  arXiv  Detail & Related papers  (2024-05-22T12:27:24Z)
• FedCAda: Adaptive Client-Side Optimization for Accelerated and Stable Federated Learning [57.38427653043984]
  Federated learning (FL) has emerged as a prominent approach for collaborative training of machine learning models across distributed clients. We introduce FedCAda, an innovative federated client adaptive algorithm designed to tackle this challenge. We demonstrate that FedCAda outperforms the state-of-the-art methods in terms of adaptability, convergence, stability, and overall performance.
  arXiv  Detail & Related papers  (2024-05-20T06:12:33Z)
• FedLALR: Client-Specific Adaptive Learning Rates Achieve Linear Speedup for Non-IID Data [54.81695390763957]
  Federated learning is an emerging distributed machine learning method. We propose a heterogeneous local variant of AMSGrad, named FedLALR, in which each client adjusts its learning rate. We show that our client-specified auto-tuned learning rate scheduling can converge and achieve linear speedup with respect to the number of clients.
  arXiv  Detail & Related papers  (2023-09-18T12:35:05Z)
• Serverless Federated AUPRC Optimization for Multi-Party Collaborative Imbalanced Data Mining [119.89373423433804]
  Area Under Precision-Recall (AUPRC) was introduced as an effective metric. Serverless multi-party collaborative training can cut down the communications cost by avoiding the server node bottleneck. We propose a new ServerLess biAsed sTochastic gradiEnt (SLATE) algorithm to directly optimize the AUPRC.
  arXiv  Detail & Related papers  (2023-08-06T06:51:32Z)
• Personalizing Federated Learning with Over-the-Air Computations [84.8089761800994]
  Federated edge learning is a promising technology to deploy intelligence at the edge of wireless networks in a privacy-preserving manner. Under such a setting, multiple clients collaboratively train a global generic model under the coordination of an edge server. This paper presents a distributed training paradigm that employs analog over-the-air computation to address the communication bottleneck.
  arXiv  Detail & Related papers  (2023-02-24T08:41:19Z)
• FilFL: Client Filtering for Optimized Client Participation in Federated Learning [71.46173076298957]
  Federated learning enables clients to collaboratively train a model without exchanging local data. Clients participating in the training process significantly impact the convergence rate, learning efficiency, and model generalization. We propose a novel approach, client filtering, to improve model generalization and optimize client participation and training.
  arXiv  Detail & Related papers  (2023-02-13T18:55:31Z)
• Communication-Efficient Federated Learning with Accelerated Client Gradient [46.81082897703729]
  Federated learning often suffers from slow and unstable convergence due to the heterogeneous characteristics of participating client datasets. We propose a simple but effective federated learning framework, which improves the consistency across clients and facilitates the convergence of the server model. We provide the theoretical convergence rate of our algorithm and demonstrate remarkable performance gains in terms of accuracy and communication efficiency.
  arXiv  Detail & Related papers  (2022-01-10T05:31:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.