General Incremental Learning with Domain-aware Categorical Representations

• URL: http://arxiv.org/abs/2204.04078v1
• Date: Fri, 8 Apr 2022 13:57:33 GMT
• Title: General Incremental Learning with Domain-aware Categorical Representations
• Authors: Jiangwei Xie, Shipeng Yan, Xuming He
• Abstract summary: We develop a novel domain-aware continual learning method based on the EM framework. Specifically, we introduce a flexible class representation based on the von Mises-Fisher mixture model to capture the intra-class structure. We design a bi-level balanced memory to cope with data imbalances within and across classes, which combines with a distillation loss to achieve better inter- and intra-class stability-plasticity trade-off.
• Score: 37.68376996568006
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Continual learning is an important problem for achieving human-level intelligence in real-world applications as an agent must continuously accumulate knowledge in response to streaming data/tasks. In this work, we consider a general and yet under-explored incremental learning problem in which both the class distribution and class-specific domain distribution change over time. In addition to the typical challenges in class incremental learning, this setting also faces the intra-class stability-plasticity dilemma and intra-class domain imbalance problems. To address above issues, we develop a novel domain-aware continual learning method based on the EM framework. Specifically, we introduce a flexible class representation based on the von Mises-Fisher mixture model to capture the intra-class structure, using an expansion-and-reduction strategy to dynamically increase the number of components according to the class complexity. Moreover, we design a bi-level balanced memory to cope with data imbalances within and across classes, which combines with a distillation loss to achieve better inter- and intra-class stability-plasticity trade-off. We conduct exhaustive experiments on three benchmarks: iDigits, iDomainNet and iCIFAR-20. The results show that our approach consistently outperforms previous methods by a significant margin, demonstrating its superiority.

Related papers

• FEED: Fairness-Enhanced Meta-Learning for Domain Generalization [13.757379847454372]
  Generalizing to out-of-distribution data while aware of model fairness is a significant and challenging problem in meta-learning. This paper introduces an approach to fairness-aware meta-learning that significantly enhances domain generalization capabilities.
  arXiv  Detail & Related papers  (2024-11-02T17:34:33Z)
• Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning [64.1745161657794]
  Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. Recent advances in pre-trained models provide a solid foundation for DIL. However, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. We propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge.
  arXiv  Detail & Related papers  (2024-10-01T17:58:06Z)
• Versatile Incremental Learning: Towards Class and Domain-Agnostic Incremental Learning [16.318126586825734]
  Incremental Learning (IL) aims to accumulate knowledge from sequential input tasks. We consider a more challenging and realistic but under-explored IL scenario, named Versatile Incremental Learning (VIL) We propose a simple yet effective IL framework, named Incremental with Shift cONtrol (ICON)
  arXiv  Detail & Related papers  (2024-09-17T07:44:28Z)
• Cross-Domain Continual Learning via CLAMP [10.553456651003055]
  CLAMP significantly outperforms established baseline algorithms across all experiments by at least $10%$ margin. An assessor-guided learning process is put forward to navigate the learning process of a base model.
  arXiv  Detail & Related papers  (2024-05-12T02:41:31Z)
• DiffClass: Diffusion-Based Class Incremental Learning [30.514281721324853]
  Class Incremental Learning (CIL) is challenging due to catastrophic forgetting. Recent exemplar-free CIL methods attempt to mitigate catastrophic forgetting by synthesizing previous task data. We propose a novel exemplar-free CIL method to overcome these issues.
  arXiv  Detail & Related papers  (2024-03-08T03:34:18Z)
• Enhancing cross-domain detection: adaptive class-aware contrastive transformer [15.666766743738531]
  Insufficient labels in the target domain exacerbate issues of class imbalance and model performance degradation. We propose a class-aware cross domain detection transformer based on the adversarial learning and mean-teacher framework.
  arXiv  Detail & Related papers  (2024-01-24T07:11:05Z)
• Neural Collapse Terminus: A Unified Solution for Class Incremental Learning and Its Variants [166.916517335816]
  In this paper, we offer a unified solution to the misalignment dilemma in the three tasks. We propose neural collapse terminus that is a fixed structure with the maximal equiangular inter-class separation for the whole label space. Our method holds the neural collapse optimality in an incremental fashion regardless of data imbalance or data scarcity.
  arXiv  Detail & Related papers  (2023-08-03T13:09:59Z)
• Learning to Augment via Implicit Differentiation for Domain Generalization [107.9666735637355]
  Domain generalization (DG) aims to overcome the problem by leveraging multiple source domains to learn a domain-generalizable model. In this paper, we propose a novel augmentation-based DG approach, dubbed AugLearn. AugLearn shows effectiveness on three standard DG benchmarks, PACS, Office-Home and Digits-DG.
  arXiv  Detail & Related papers  (2022-10-25T18:51:51Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Dynamic Federated Learning [57.14673504239551]
  Federated learning has emerged as an umbrella term for centralized coordination strategies in multi-agent environments. We consider a federated learning model where at every iteration, a random subset of available agents perform local updates based on their data. Under a non-stationary random walk model on the true minimizer for the aggregate optimization problem, we establish that the performance of the architecture is determined by three factors, namely, the data variability at each agent, the model variability across all agents, and a tracking term that is inversely proportional to the learning rate of the algorithm.
  arXiv  Detail & Related papers  (2020-02-20T15:00:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.