Masked Autoencoders are Efficient Continual Federated Learners

• URL: http://arxiv.org/abs/2306.03542v2
• Date: Thu, 18 Jul 2024 13:22:21 GMT
• Title: Masked Autoencoders are Efficient Continual Federated Learners
• Authors: Subarnaduti Paul, Lars-Joel Frey, Roshni Kamath, Kristian Kersting, Martin Mundt,
• Abstract summary: Continual learning should be grounded in unsupervised learning of representations that are shared across clients. Masked autoencoders for distribution estimation are particularly amenable to this setup.
• Score: 20.856520787551453
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine learning is typically framed from a perspective of i.i.d., and more importantly, isolated data. In parts, federated learning lifts this assumption, as it sets out to solve the real-world challenge of collaboratively learning a shared model from data distributed across clients. However, motivated primarily by privacy and computational constraints, the fact that data may change, distributions drift, or even tasks advance individually on clients, is seldom taken into account. The field of continual learning addresses this separate challenge and first steps have recently been taken to leverage synergies in distributed supervised settings, in which several clients learn to solve changing classification tasks over time without forgetting previously seen ones. Motivated by these prior works, we posit that such federated continual learning should be grounded in unsupervised learning of representations that are shared across clients; in the loose spirit of how humans can indirectly leverage others' experience without exposure to a specific task. For this purpose, we demonstrate that masked autoencoders for distribution estimation are particularly amenable to this setup. Specifically, their masking strategy can be seamlessly integrated with task attention mechanisms to enable selective knowledge transfer between clients. We empirically corroborate the latter statement through several continual federated scenarios on both image and binary datasets.

Related papers

• Personalized Federated Learning via Amortized Bayesian Meta-Learning [21.126405589760367]
  We introduce a new perspective on personalized federated learning through Amortized Bayesian Meta-Learning. Specifically, we propose a novel algorithm called emphFedABML, which employs hierarchical variational inference across clients. Our theoretical analysis provides an upper bound on the average generalization error and guarantees the generalization performance on unseen data.
  arXiv  Detail & Related papers  (2023-07-05T11:58:58Z)
• Leveraging sparse and shared feature activations for disentangled representation learning [112.22699167017471]
  We propose to leverage knowledge extracted from a diversified set of supervised tasks to learn a common disentangled representation. We validate our approach on six real world distribution shift benchmarks, and different data modalities.
  arXiv  Detail & Related papers  (2023-04-17T01:33:24Z)
• FedClassAvg: Local Representation Learning for Personalized Federated Learning on Heterogeneous Neural Networks [21.613436984547917]
  We propose a novel personalized federated learning method called federated classifier averaging (FedClassAvg) FedClassAvg aggregates weights as an agreement on decision boundaries on feature spaces. We demonstrate it outperforms the current state-of-the-art algorithms on heterogeneous personalized federated learning tasks.
  arXiv  Detail & Related papers  (2022-10-25T08:32:08Z)
• Straggler-Resilient Personalized Federated Learning [55.54344312542944]
  Federated learning allows training models from samples distributed across a large network of clients while respecting privacy and communication restrictions. We develop a novel algorithmic procedure with theoretical speedup guarantees that simultaneously handles two of these hurdles. Our method relies on ideas from representation learning theory to find a global common representation using all clients' data and learn a user-specific set of parameters leading to a personalized solution for each client.
  arXiv  Detail & Related papers  (2022-06-05T01:14:46Z)
• Federated Self-supervised Learning for Heterogeneous Clients [20.33482170846688]
  We propose a unified and systematic framework, emphHeterogeneous Self-supervised Federated Learning (Hetero-SSFL) for enabling self-supervised learning with federation on heterogeneous clients. The proposed framework allows representation learning across all the clients without imposing architectural constraints or requiring presence of labeled data. We empirically demonstrate that our proposed approach outperforms the state of the art methods by a significant margin.
  arXiv  Detail & Related papers  (2022-05-25T05:07:44Z)
• Self-Supervised Graph Neural Network for Multi-Source Domain Adaptation [51.21190751266442]
  Domain adaptation (DA) tries to tackle the scenarios when the test data does not fully follow the same distribution of the training data. By learning from large-scale unlabeled samples, self-supervised learning has now become a new trend in deep learning. We propose a novel textbfSelf-textbfSupervised textbfGraph Neural Network (SSG) to enable more effective inter-task information exchange and knowledge sharing.
  arXiv  Detail & Related papers  (2022-04-08T03:37:56Z)
• Addressing Client Drift in Federated Continual Learning with Adaptive Optimization [10.303676184878896]
  We outline a framework for performing Federated Continual Learning (FCL) by using NetTailor as a candidate continual learning approach. We show that adaptive federated optimization can reduce the adverse impact of client drift and showcase its effectiveness on CIFAR100, MiniImagenet, and Decathlon benchmarks.
  arXiv  Detail & Related papers  (2022-03-24T20:00:03Z)
• Tackling Dynamics in Federated Incremental Learning with Variational Embedding Rehearsal [27.64806509651952]
  We propose a novel algorithm to address the incremental learning process in an FL scenario. We first propose using deep Variational Embeddings that secure the privacy of the client data. Second, we propose a server-side training method that enables a model to rehearse the previously learnt knowledge.
  arXiv  Detail & Related papers  (2021-10-19T02:26:35Z)
• Federated Multi-Target Domain Adaptation [99.93375364579484]
  Federated learning methods enable us to train machine learning models on distributed user data while preserving its privacy. We consider a more practical scenario where the distributed client data is unlabeled, and a centralized labeled dataset is available on the server. We propose an effective DualAdapt method to address the new challenges.
  arXiv  Detail & Related papers  (2021-08-17T17:53:05Z)
• Exploiting Shared Representations for Personalized Federated Learning [54.65133770989836]
  We propose a novel federated learning framework and algorithm for learning a shared data representation across clients and unique local heads for each client. Our algorithm harnesses the distributed computational power across clients to perform many local-updates with respect to the low-dimensional local parameters for every update of the representation. This result is of interest beyond federated learning to a broad class of problems in which we aim to learn a shared low-dimensional representation among data distributions.
  arXiv  Detail & Related papers  (2021-02-14T05:36:25Z)
• Federated Continual Learning with Weighted Inter-client Transfer [79.93004004545736]
  We propose a novel federated continual learning framework, Federated Weighted Inter-client Transfer (FedWeIT) FedWeIT decomposes the network weights into global federated parameters and sparse task-specific parameters, and each client receives selective knowledge from other clients. We validate our FedWeIT against existing federated learning and continual learning methods, and our model significantly outperforms them with a large reduction in the communication cost.
  arXiv  Detail & Related papers  (2020-03-06T13:33:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.