Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning

• URL: http://arxiv.org/abs/2410.00911v1
• Date: Tue, 1 Oct 2024 17:58:06 GMT
• Title: Dual Consolidation for Pre-Trained Model-Based Domain-Incremental Learning
• Authors: Da-Wei Zhou, Zi-Wen Cai, Han-Jia Ye, Lijun Zhang, De-Chuan Zhan,
• Abstract summary: Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. Recent advances in pre-trained models provide a solid foundation for DIL. However, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. We propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge.
• Score: 64.1745161657794
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Domain-Incremental Learning (DIL) involves the progressive adaptation of a model to new concepts across different domains. While recent advances in pre-trained models provide a solid foundation for DIL, learning new concepts often results in the catastrophic forgetting of pre-trained knowledge. Specifically, sequential model updates can overwrite both the representation and the classifier with knowledge from the latest domain. Thus, it is crucial to develop a representation and corresponding classifier that accommodate all seen domains throughout the learning process. To this end, we propose DUal ConsolidaTion (Duct) to unify and consolidate historical knowledge at both the representation and classifier levels. By merging the backbone of different stages, we create a representation space suitable for multiple domains incrementally. The merged representation serves as a balanced intermediary that captures task-specific features from all seen domains. Additionally, to address the mismatch between consolidated embeddings and the classifier, we introduce an extra classifier consolidation process. Leveraging class-wise semantic information, we estimate the classifier weights of old domains within the latest embedding space. By merging historical and estimated classifiers, we align them with the consolidated embedding space, facilitating incremental classification. Extensive experimental results on four benchmark datasets demonstrate Duct's state-of-the-art performance.

Related papers

• Read Between the Layers: Leveraging Multi-Layer Representations for Rehearsal-Free Continual Learning with Pre-Trained Models [15.847302755988506]
  We address the Continual Learning problem, wherein a model must learn a sequence of tasks from non-stationary distributions. We propose LayUP, a new prototype-based approach to CL that leverages second-order feature statistics from multiple intermediate layers of a pre-trained network. Our results demonstrate that fully exhausting the representational capacities of pre-trained models in CL goes well beyond their final embeddings.
  arXiv  Detail & Related papers  (2023-12-13T13:11:44Z)
• Multi-View Class Incremental Learning [57.14644913531313]
  Multi-view learning (MVL) has gained great success in integrating information from multiple perspectives of a dataset to improve downstream task performance. This paper investigates a novel paradigm called multi-view class incremental learning (MVCIL), where a single model incrementally classifies new classes from a continual stream of views.
  arXiv  Detail & Related papers  (2023-06-16T08:13:41Z)
• Incremental Few-Shot Semantic Segmentation via Embedding Adaptive-Update and Hyper-class Representation [30.558312809285905]
  EHNet achieves new state-of-the-art performance with remarkable advantages. Experiments on PASCAL-5i and COCO datasets show that EHNet achieves new state-of-the-art performance with remarkable advantages.
  arXiv  Detail & Related papers  (2022-07-26T15:20:07Z)
• Forget Less, Count Better: A Domain-Incremental Self-Distillation Learning Benchmark for Lifelong Crowd Counting [51.44987756859706]
  Off-the-shelf methods have some drawbacks to handle multiple domains. Lifelong Crowd Counting aims at alleviating the catastrophic forgetting and improving the generalization ability.
  arXiv  Detail & Related papers  (2022-05-06T15:37:56Z)
• HRKD: Hierarchical Relational Knowledge Distillation for Cross-domain Language Model Compression [53.90578309960526]
  Large pre-trained language models (PLMs) have shown overwhelming performances compared with traditional neural network methods. We propose a hierarchical relational knowledge distillation (HRKD) method to capture both hierarchical and domain relational information.
  arXiv  Detail & Related papers  (2021-10-16T11:23:02Z)
• Context-Conditional Adaptation for Recognizing Unseen Classes in Unseen Domains [48.17225008334873]
  We propose a feature generative framework integrated with a COntext COnditional Adaptive (COCOA) Batch-Normalization. The generated visual features better capture the underlying data distribution enabling us to generalize to unseen classes and domains at test-time. We thoroughly evaluate and analyse our approach on established large-scale benchmark - DomainNet.
  arXiv  Detail & Related papers  (2021-07-15T17:51:16Z)
• Universal Representation Learning from Multiple Domains for Few-shot Classification [41.821234589075445]
  We propose to learn a single set of universal deep representations by distilling knowledge of multiple separately trained networks. We show that the universal representations can be further refined for previously unseen domains by an efficient adaptation step.
  arXiv  Detail & Related papers  (2021-03-25T13:49:12Z)
• Learning to Combine: Knowledge Aggregation for Multi-Source Domain Adaptation [56.694330303488435]
  We propose a Learning to Combine for Multi-Source Domain Adaptation (LtC-MSDA) framework. In the nutshell, a knowledge graph is constructed on the prototypes of various domains to realize the information propagation among semantically adjacent representations. Our approach outperforms existing methods with a remarkable margin.
  arXiv  Detail & Related papers  (2020-07-17T07:52:44Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.