Deep Class Incremental Learning from Decentralized Data

• URL: http://arxiv.org/abs/2203.05984v1
• Date: Fri, 11 Mar 2022 15:09:33 GMT
• Title: Deep Class Incremental Learning from Decentralized Data
• Authors: Xiaohan Zhang, Songlin Dong, Jinjie Chen, Qi Tian, Yihong Gong, Xiaopeng Hong
• Abstract summary: We focus on a new and challenging decentralized machine learning paradigm in which there are continuous inflows of data to be addressed. We introduce a paradigm to create a basic decentralized counterpart of typical (centralized) class-incremental learning approaches. We propose a Decentralized Composite knowledge Incremental Distillation framework (DCID) to transfer knowledge from historical models and multiple local sites to the general model continually.
• Score: 103.2386956343121
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we focus on a new and challenging decentralized machine learning paradigm in which there are continuous inflows of data to be addressed and the data are stored in multiple repositories. We initiate the study of data decentralized class-incremental learning (DCIL) by making the following contributions. Firstly, we formulate the DCIL problem and develop the experimental protocol. Secondly, we introduce a paradigm to create a basic decentralized counterpart of typical (centralized) class-incremental learning approaches, and as a result, establish a benchmark for the DCIL study. Thirdly, we further propose a Decentralized Composite knowledge Incremental Distillation framework (DCID) to transfer knowledge from historical models and multiple local sites to the general model continually. DCID consists of three main components namely local class-incremental learning, collaborated knowledge distillation among local models, and aggregated knowledge distillation from local models to the general one. We comprehensively investigate our DCID framework by using different implementations of the three components. Extensive experimental results demonstrate the effectiveness of our DCID framework. The codes of the baseline methods and the proposed DCIL will be released at https://github.com/zxxxxh/DCIL.

Related papers

• Distribution Shift Matters for Knowledge Distillation with Webly Collected Images [91.66661969598755]
  We propose a novel method dubbed Knowledge Distillation between Different Distributions" (KD$3$) We first dynamically select useful training instances from the webly collected data according to the combined predictions of teacher network and student network. We also build a new contrastive learning block called MixDistribution to generate perturbed data with a new distribution for instance alignment.
  arXiv  Detail & Related papers  (2023-07-21T10:08:58Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• CMD: Self-supervised 3D Action Representation Learning with Cross-modal Mutual Distillation [130.08432609780374]
  In 3D action recognition, there exists rich complementary information between skeleton modalities. We propose a new Cross-modal Mutual Distillation (CMD) framework with the following designs. Our approach outperforms existing self-supervised methods and sets a series of new records.
  arXiv  Detail & Related papers  (2022-08-26T06:06:09Z)
• Point-to-Voxel Knowledge Distillation for LiDAR Semantic Segmentation [74.67594286008317]
  This article addresses the problem of distilling knowledge from a large teacher model to a slim student network for LiDAR semantic segmentation. We propose the Point-to-Voxel Knowledge Distillation (PVD), which transfers the hidden knowledge from both point level and voxel level.
  arXiv  Detail & Related papers  (2022-06-05T05:28:32Z)
• FedDKD: Federated Learning with Decentralized Knowledge Distillation [3.9084449541022055]
  We propose a novel framework of federated learning equipped with the process of decentralized knowledge distillation (FedDKD) We show that FedDKD outperforms the state-of-the-art methods with more efficient communication and training in a few DKD steps.
  arXiv  Detail & Related papers  (2022-05-02T07:54:07Z)
• Static-Dynamic Co-Teaching for Class-Incremental 3D Object Detection [71.18882803642526]
  Deep learning approaches have shown remarkable performance in the 3D object detection task. They suffer from a catastrophic performance drop when incrementally learning new classes without revisiting the old data. This "catastrophic forgetting" phenomenon impedes the deployment of 3D object detection approaches in real-world scenarios. We present the first solution - SDCoT, a novel static-dynamic co-teaching method.
  arXiv  Detail & Related papers  (2021-12-14T09:03:41Z)
• Cross-Gradient Aggregation for Decentralized Learning from Non-IID data [34.23789472226752]
  Decentralized learning enables a group of collaborative agents to learn models using a distributed dataset without the need for a central parameter server. We propose Cross-Gradient Aggregation (CGA), a novel decentralized learning algorithm. We show superior learning performance of CGA over existing state-of-the-art decentralized learning algorithms.
  arXiv  Detail & Related papers  (2021-03-02T21:58:12Z)
• Adaptive Distillation for Decentralized Learning from Heterogeneous Clients [9.261720698142097]
  We propose a new decentralized learning method called Decentralized Learning via Adaptive Distillation (DLAD) The proposed DLAD aggregates the outputs of the client models while adaptively emphasizing those with higher confidence in given distillation samples. Our extensive experimental evaluation on multiple public datasets demonstrates the effectiveness of the proposed method.
  arXiv  Detail & Related papers  (2020-08-18T14:25:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.