FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning

• URL: http://arxiv.org/abs/2410.06490v1
• Date: Wed, 9 Oct 2024 02:31:49 GMT
• Title: FedL2G: Learning to Guide Local Training in Heterogeneous Federated Learning
• Authors: Jianqing Zhang, Yang Liu, Yang Hua, Jian Cao, Qiang Yang,
• Abstract summary: In Heterogeneous Federated Learning (HtFL) scenarios, aggregating model parameters leads to the use of prototypes for aggregation and guidance. We propose a training-to-Guide (FedL2G) method that adaptively learns to guide local settings in a manner that ensures extra guidance is beneficial to clients' original settings.
• Score: 23.92461217732838
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Data and model heterogeneity are two core issues in Heterogeneous Federated Learning (HtFL). In scenarios with heterogeneous model architectures, aggregating model parameters becomes infeasible, leading to the use of prototypes (i.e., class representative feature vectors) for aggregation and guidance. However, they still experience a mismatch between the extra guiding objective and the client's original local objective when aligned with global prototypes. Thus, we propose a Federated Learning-to-Guide (FedL2G) method that adaptively learns to guide local training in a federated manner and ensures the extra guidance is beneficial to clients' original tasks. With theoretical guarantees, FedL2G efficiently implements the learning-to-guide process using only first-order derivatives w.r.t. model parameters and achieves a non-convex convergence rate of O(1/T). We conduct extensive experiments on two data heterogeneity and six model heterogeneity settings using 14 heterogeneous model architectures (e.g., CNNs and ViTs) to demonstrate FedL2G's superior performance compared to six counterparts.

Related papers

• Federated Model Heterogeneous Matryoshka Representation Learning [33.04969829305812]
  Model heterogeneous federated learning (MteroFL) enables FL clients to collaboratively train models with heterogeneous structures in a distributed fashion. Existing methods rely on training loss to transfer knowledge between a MteroFL server and a client model. We propose a new representation approach for supervised learning tasks using Matryoshka models.
  arXiv  Detail & Related papers  (2024-06-01T16:37:08Z)
• Task Groupings Regularization: Data-Free Meta-Learning with Heterogeneous Pre-trained Models [83.02797560769285]
  Data-Free Meta-Learning (DFML) aims to derive knowledge from a collection of pre-trained models without accessing their original data. Current methods often overlook the heterogeneity among pre-trained models, which leads to performance degradation due to task conflicts. We propose Task Groupings Regularization, a novel approach that benefits from model heterogeneity by grouping and aligning conflicting tasks.
  arXiv  Detail & Related papers  (2024-05-26T13:11:55Z)
• pFedAFM: Adaptive Feature Mixture for Batch-Level Personalization in Heterogeneous Federated Learning [34.01721941230425]
  We propose a model-heterogeneous personalized Federated learning approach with Adaptive Feature Mixture (pFedAFM) for supervised learning tasks. It significantly outperforms 7 state-of-the-art MHPFL methods, achieving up to 7.93% accuracy improvement.
  arXiv  Detail & Related papers  (2024-04-27T09:52:59Z)
• Robust Training of Federated Models with Extremely Label Deficiency [84.00832527512148]
  Federated semi-supervised learning (FSSL) has emerged as a powerful paradigm for collaboratively training machine learning models using distributed data with label deficiency. We propose a novel twin-model paradigm, called Twin-sight, designed to enhance mutual guidance by providing insights from different perspectives of labeled and unlabeled data. Our comprehensive experiments on four benchmark datasets provide substantial evidence that Twin-sight can significantly outperform state-of-the-art methods across various experimental settings.
  arXiv  Detail & Related papers  (2024-02-22T10:19:34Z)
• Federated Learning with Projected Trajectory Regularization [65.6266768678291]
  Federated learning enables joint training of machine learning models from distributed clients without sharing their local data. One key challenge in federated learning is to handle non-identically distributed data across the clients. We propose a novel federated learning framework with projected trajectory regularization (FedPTR) for tackling the data issue.
  arXiv  Detail & Related papers  (2023-12-22T02:12:08Z)
• Fake It Till Make It: Federated Learning with Consensus-Oriented Generation [52.82176415223988]
  We propose federated learning with consensus-oriented generation (FedCOG) FedCOG consists of two key components at the client side: complementary data generation and knowledge-distillation-based model training. Experiments on classical and real-world FL datasets show that FedCOG consistently outperforms state-of-the-art methods.
  arXiv  Detail & Related papers  (2023-12-10T18:49:59Z)
• Enhancing Representations through Heterogeneous Self-Supervised Learning [61.40674648939691]
  We propose Heterogeneous Self-Supervised Learning (HSSL), which enforces a base model to learn from an auxiliary head whose architecture is heterogeneous from the base model. The HSSL endows the base model with new characteristics in a representation learning way without structural changes. The HSSL is compatible with various self-supervised methods, achieving superior performances on various downstream tasks.
  arXiv  Detail & Related papers  (2023-10-08T10:44:05Z)
• Prototype Helps Federated Learning: Towards Faster Convergence [38.517903009319994]
  Federated learning (FL) is a distributed machine learning technique in which multiple clients cooperate to train a shared model without exchanging their raw data. In this paper, a prototype-based federated learning framework is proposed, which can achieve better inference performance with only a few changes to the last global iteration of the typical federated learning process.
  arXiv  Detail & Related papers  (2023-03-22T04:06:29Z)
• Faster Adaptive Federated Learning [84.38913517122619]
  Federated learning has attracted increasing attention with the emergence of distributed data. In this paper, we propose an efficient adaptive algorithm (i.e., FAFED) based on momentum-based variance reduced technique in cross-silo FL.
  arXiv  Detail & Related papers  (2022-12-02T05:07:50Z)
• FedSiam-DA: Dual-aggregated Federated Learning via Siamese Network under Non-IID Data [21.95009868875851]
  Federated learning can address data island, it remains challenging to train with data heterogeneous in a real application. We propose FedSiam-DA, a novel dual-aggregated contrastive federated learning approach.
  arXiv  Detail & Related papers  (2022-11-17T09:05:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.