Recurrent Early Exits for Federated Learning with Heterogeneous Clients

• URL: http://arxiv.org/abs/2405.14791v2
• Date: Mon, 27 May 2024 16:11:22 GMT
• Title: Recurrent Early Exits for Federated Learning with Heterogeneous Clients
• Authors: Royson Lee, Javier Fernandez-Marques, Shell Xu Hu, Da Li, Stefanos Laskaridis, Łukasz Dudziak, Timothy Hospedales, Ferenc Huszár, Nicholas D. Lane,
• Abstract summary: Federated learning (FL) has enabled distributed learning of a model across multiple clients in a privacy-preserving manner. One of the main challenges of FL is to accommodate clients with varying hardware capacities. We propose a recurrent early exit approach named ReeFL that fuses features from different sub-models into a single shared classifier.
• Score: 22.429334632124817
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated learning (FL) has enabled distributed learning of a model across multiple clients in a privacy-preserving manner. One of the main challenges of FL is to accommodate clients with varying hardware capacities; clients have differing compute and memory requirements. To tackle this challenge, recent state-of-the-art approaches leverage the use of early exits. Nonetheless, these approaches fall short of mitigating the challenges of joint learning multiple exit classifiers, often relying on hand-picked heuristic solutions for knowledge distillation among classifiers and/or utilizing additional layers for weaker classifiers. In this work, instead of utilizing multiple classifiers, we propose a recurrent early exit approach named ReeFL that fuses features from different sub-models into a single shared classifier. Specifically, we use a transformer-based early-exit module shared among sub-models to i) better exploit multi-layer feature representations for task-specific prediction and ii) modulate the feature representation of the backbone model for subsequent predictions. We additionally present a per-client self-distillation approach where the best sub-model is automatically selected as the teacher of the other sub-models at each client. Our experiments on standard image and speech classification benchmarks across various emerging federated fine-tuning baselines demonstrate ReeFL's effectiveness over previous works.

Related papers

• Not All Clients Are Equal: Personalized Federated Learning on Heterogeneous Multi-Modal Clients [52.14230635007546]
  Foundation models have shown remarkable capabilities across diverse multi-modal tasks, but their centralized training raises privacy concerns and induces high transmission costs.<n>For the growing demand for personalizing AI models for different user purposes, personalized federated learning (PFL) has emerged.<n>PFL allows each client to leverage the knowledge of other clients for further adaptation to individual user preferences, again without the need to share data.
  arXiv  Detail & Related papers  (2025-05-20T09:17:07Z)
• FeNeC: Enhancing Continual Learning via Feature Clustering with Neighbor- or Logit-Based Classification [6.720605329045581]
  We introduce FeNeC (Feature Neighborhood) and FeNeC-Log, incorporating its variant based on the log-likelihood function. Our approach generalizes the existing concept by clustering to capture greater intra-class variability. We demonstrate that two FeNeC variants achieve competitive performance in scenarios where task identities are unknown.
  arXiv  Detail & Related papers  (2025-03-18T14:42:38Z)
• Look Back for More: Harnessing Historical Sequential Updates for Personalized Federated Adapter Tuning [50.45027483522507]
  Existing personalized federated learning (PFL) approaches rely solely on the clients' latest updated models. We propose pFedSeq, designed for personalizing adapters to fine-tune a foundation model in FL. In pFedSeq, the server maintains and trains a sequential learner, which processes a sequence of past adapter updates from clients. To effectively capture the cross-client and cross-step relations hidden in previous updates, pFedSeq adopts the powerful selective state space model.
  arXiv  Detail & Related papers  (2025-01-03T06:10:09Z)
• Vertical Federated Unlearning via Backdoor Certification [15.042986414487922]
  VFL offers a novel paradigm in machine learning, enabling distinct entities to train models cooperatively while maintaining data privacy. Recent privacy regulations emphasize an individual's emphright to be forgotten, which necessitates the ability for models to unlearn specific training data. We introduce an innovative modification to traditional VFL by employing a mechanism that inverts the typical learning trajectory with the objective of extracting specific data contributions.
  arXiv  Detail & Related papers  (2024-12-16T06:40:25Z)
• Personalized Hierarchical Split Federated Learning in Wireless Networks [24.664469755746463]
  We propose a personalized hierarchical split federated learning (PHSFL) algorithm that is specially designed to achieve better personalization performance. We first perform extensive theoretical analysis to understand the impact of model splitting and hierarchical model aggregations on the global model. Once the global model is trained, we fine-tune each client to obtain the personalized models.
  arXiv  Detail & Related papers  (2024-11-09T02:41:53Z)
• Revisiting SMoE Language Models by Evaluating Inefficiencies with Task Specific Expert Pruning [78.72226641279863]
  Sparse Mixture of Expert (SMoE) models have emerged as a scalable alternative to dense models in language modeling. Our research explores task-specific model pruning to inform decisions about designing SMoE architectures. We introduce an adaptive task-aware pruning technique UNCURL to reduce the number of experts per MoE layer in an offline manner post-training.
  arXiv  Detail & Related papers  (2024-09-02T22:35:03Z)
• Multi-Level Additive Modeling for Structured Non-IID Federated Learning [54.53672323071204]
  We train models organized in a multi-level structure, called Multi-level Additive Models (MAM)'', for better knowledge-sharing across heterogeneous clients. In federated MAM (FeMAM), each client is assigned to at most one model per level and its personalized prediction sums up the outputs of models assigned to it across all levels. Experiments show that FeMAM surpasses existing clustered FL and personalized FL methods in various non-IID settings.
  arXiv  Detail & Related papers  (2024-05-26T07:54:53Z)
• FeCAM: Exploiting the Heterogeneity of Class Distributions in Exemplar-Free Continual Learning [21.088762527081883]
  Exemplar-free class-incremental learning (CIL) poses several challenges since it prohibits the rehearsal of data from previous tasks. Recent approaches to incrementally learning the classifier by freezing the feature extractor after the first task have gained much attention. We explore prototypical networks for CIL, which generate new class prototypes using the frozen feature extractor and classify the features based on the Euclidean distance to the prototypes.
  arXiv  Detail & Related papers  (2023-09-25T11:54:33Z)
• RanPAC: Random Projections and Pre-trained Models for Continual Learning [59.07316955610658]
  Continual learning (CL) aims to learn different tasks (such as classification) in a non-stationary data stream without forgetting old ones. We propose a concise and effective approach for CL with pre-trained models.
  arXiv  Detail & Related papers  (2023-07-05T12:49:02Z)
• Complementary Learning Subnetworks for Parameter-Efficient Class-Incremental Learning [40.13416912075668]
  We propose a rehearsal-free CIL approach that learns continually via the synergy between two Complementary Learning Subnetworks. Our method achieves competitive results against state-of-the-art methods, especially in accuracy gain, memory cost, training efficiency, and task-order.
  arXiv  Detail & Related papers  (2023-06-21T01:43:25Z)
• Multi-View Class Incremental Learning [57.14644913531313]
  Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance. This paper investigates a novel paradigm called multi-view class incremental learning (MVCIL), where a single model incrementally classifies new classes from a continual stream of views.
  arXiv  Detail & Related papers  (2023-06-16T08:13:41Z)
• Enhancing Continual Relation Extraction via Classifier Decomposition [30.88081408988638]
  Continual relation extraction models aim at handling emerging new relations while avoiding forgetting old ones in the streaming data. Most models only adopt a vanilla strategy when models first learn representations of new relations. We propose a simple yet effective classifier decomposition framework that splits the last FFN layer into separated previous and current classifiers.
  arXiv  Detail & Related papers  (2023-05-08T11:29:33Z)
• Federated Learning from Pre-Trained Models: A Contrastive Learning Approach [43.893267526525904]
  Federated Learning (FL) is a machine learning paradigm that allows decentralized clients to learn collaboratively without sharing their private data. Excessive computation and communication demands pose challenges to current FL frameworks. We propose a lightweight framework where clients jointly learn to fuse the representations generated by multiple fixed pre-trained models.
  arXiv  Detail & Related papers  (2022-09-21T03:16:57Z)
• Exploring Complementary Strengths of Invariant and Equivariant Representations for Few-Shot Learning [96.75889543560497]
  In many real-world problems, collecting a large number of labeled samples is infeasible. Few-shot learning is the dominant approach to address this issue, where the objective is to quickly adapt to novel categories in presence of a limited number of samples. We propose a novel training mechanism that simultaneously enforces equivariance and invariance to a general set of geometric transformations.
  arXiv  Detail & Related papers  (2021-03-01T21:14:33Z)
• UVeQFed: Universal Vector Quantization for Federated Learning [179.06583469293386]
  Federated learning (FL) is an emerging approach to train such learning models without requiring the users to share their possibly private labeled data. In FL, each user trains its copy of the learning model locally. The server then collects the individual updates and aggregates them into a global model. We show that combining universal vector quantization methods with FL yields a decentralized training system in which the compression of the trained models induces only a minimum distortion.
  arXiv  Detail & Related papers  (2020-06-05T07:10:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.