Addressing Spatial-Temporal Data Heterogeneity in Federated Continual Learning via Tail Anchor

• URL: http://arxiv.org/abs/2412.18355v1
• Date: Tue, 24 Dec 2024 11:35:40 GMT
• Title: Addressing Spatial-Temporal Data Heterogeneity in Federated Continual Learning via Tail Anchor
• Authors: Hao Yu, Xin Yang, Le Zhang, Hanlin Gu, Tianrui Li, Lixin Fan, Qiang Yang,
• Abstract summary: Federated continual learning (FCL) allows each client to continually update its knowledge from task streams. We propose Federated Tail Anchor (FedTA) to mix trainable Tail Anchor with the frozen output features to adjust their position in the feature space. FedTA not only outperforms existing FCL methods but also effectively preserves the relative positions of features, remaining unaffected by spatial and temporal changes.
• Score: 24.689188066180463
• License:
• Abstract: Federated continual learning (FCL) allows each client to continually update its knowledge from task streams, enhancing the applicability of federated learning in real-world scenarios. However, FCL needs to address not only spatial data heterogeneity between clients but also temporal data heterogeneity between tasks. In this paper, empirical experiments demonstrate that such input-level heterogeneity significantly affects the model's internal parameters and outputs, leading to severe spatial-temporal catastrophic forgetting of local and previous knowledge. To this end, we propose Federated Tail Anchor (FedTA) to mix trainable Tail Anchor with the frozen output features to adjust their position in the feature space, thereby overcoming parameter-forgetting and output-forgetting. Moreover, three novel components are also included in FedTA: Input Enhancement for improving the performance of pre-trained models on downstream tasks; Selective Input Knowledge Fusion for fusion of heterogeneous local knowledge on the server side; and Best Global Prototype Selection for finding the best anchor point for each class in the feature space. Extensive experiments demonstrate that FedTA not only outperforms existing FCL methods but also effectively preserves the relative positions of features, remaining unaffected by spatial and temporal changes.

Related papers

• FedLF: Adaptive Logit Adjustment and Feature Optimization in Federated Long-Tailed Learning [5.23984567704876]
  Federated learning offers a paradigm to the challenge of preserving privacy in distributed machine learning. Traditional approach fails to address the phenomenon of class-wise bias in global long-tailed data. New method FedLF introduces three modifications in the local training phase: adaptive logit adjustment, continuous class centred optimization, and feature decorrelation.
  arXiv  Detail & Related papers  (2024-09-18T16:25:29Z)
• Federated Learning under Partially Class-Disjoint Data via Manifold Reshaping [64.58402571292723]
  We propose a manifold reshaping approach called FedMR to calibrate the feature space of local training. We conduct extensive experiments on a range of datasets to demonstrate that our FedMR achieves much higher accuracy and better communication efficiency.
  arXiv  Detail & Related papers  (2024-05-29T10:56:13Z)
• Decoupled Federated Learning on Long-Tailed and Non-IID data with Feature Statistics [20.781607752797445]
  We propose a two-stage Decoupled Federated learning framework using Feature Statistics (DFL-FS) In the first stage, the server estimates the client's class coverage distributions through masked local feature statistics clustering. In the second stage, DFL-FS employs federated feature regeneration based on global feature statistics to enhance the model's adaptability to long-tailed data distributions.
  arXiv  Detail & Related papers  (2024-03-13T09:24:59Z)
• FedFed: Feature Distillation against Data Heterogeneity in Federated Learning [88.36513907827552]
  Federated learning (FL) typically faces data heterogeneity, i.e., distribution shifting among clients. We propose a novel approach called textbfFederated textbfFeature textbfdistillation (FedFedFed) FedFed partitions data into performance-sensitive features (i.e., greatly contributing to model performance) and performance-robust features (i.e., limitedly contributing to model performance) Comprehensive experiments demonstrate the efficacy of FedFed in promoting model performance.
  arXiv  Detail & Related papers  (2023-10-08T09:00:59Z)
• Neural Collapse Inspired Federated Learning with Non-iid Data [31.576588815816095]
  Non-independent and identically distributed (non-iid) characteristics cause significant differences in local updates and affect the performance of the central server. Inspired by the phenomenon of neural collapse, we force each client to be optimized toward an optimal global structure for classification. Our method can improve the performance with faster convergence speed on different-size datasets.
  arXiv  Detail & Related papers  (2023-03-27T05:29:53Z)
• FedFA: Federated Learning with Feature Anchors to Align Features and Classifiers for Heterogeneous Data [8.677832361022809]
  Federated learning allows multiple clients to collaboratively train a model without exchanging their data. Common solutions involve an auxiliary loss to regularize weight divergence or feature inconsistency during local training. We propose a novel framework named Federated learning with Feature Anchors (FedFA)
  arXiv  Detail & Related papers  (2022-11-17T02:27:44Z)
• FedFM: Anchor-based Feature Matching for Data Heterogeneity in Federated Learning [91.74206675452888]
  We propose a novel method FedFM, which guides each client's features to match shared category-wise anchors. To achieve higher efficiency and flexibility, we propose a FedFM variant, called FedFM-Lite, where clients communicate with server with fewer synchronization times and communication bandwidth costs.
  arXiv  Detail & Related papers  (2022-10-14T08:11:34Z)
• Rethinking Data Heterogeneity in Federated Learning: Introducing a New Notion and Standard Benchmarks [65.34113135080105]
  We show that not only the issue of data heterogeneity in current setups is not necessarily a problem but also in fact it can be beneficial for the FL participants. Our observations are intuitive. Our code is available at https://github.com/MMorafah/FL-SC-NIID.
  arXiv  Detail & Related papers  (2022-09-30T17:15:19Z)
• FedDC: Federated Learning with Non-IID Data via Local Drift Decoupling and Correction [48.85303253333453]
  Federated learning (FL) allows multiple clients to collectively train a high-performance global model without sharing their private data. We propose a novel federated learning algorithm with local drift decoupling and correction (FedDC) Our FedDC only introduces lightweight modifications in the local training phase, in which each client utilizes an auxiliary local drift variable to track the gap between the local model parameter and the global model parameters. Experiment results and analysis demonstrate that FedDC yields expediting convergence and better performance on various image classification tasks.
  arXiv  Detail & Related papers  (2022-03-22T14:06:26Z)
• Acceleration of Federated Learning with Alleviated Forgetting in Local Training [61.231021417674235]
  Federated learning (FL) enables distributed optimization of machine learning models while protecting privacy. We propose FedReg, an algorithm to accelerate FL with alleviated knowledge forgetting in the local training stage. Our experiments demonstrate that FedReg not only significantly improves the convergence rate of FL, especially when the neural network architecture is deep.
  arXiv  Detail & Related papers  (2022-03-05T02:31:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.