Class-relation Knowledge Distillation for Novel Class Discovery
- URL: http://arxiv.org/abs/2307.09158v3
- Date: Fri, 25 Aug 2023 13:59:08 GMT
- Title: Class-relation Knowledge Distillation for Novel Class Discovery
- Authors: Peiyan Gu, Chuyu Zhang, Ruijie Xu, Xuming He
- Abstract summary: Key challenge lies in transferring the knowledge in the known-class data to the learning of novel classes.
We introduce a class relation representation for the novel classes based on the predicted class distribution of a model trained on known classes.
We propose a novel knowledge distillation framework, which utilizes our class-relation representation to regularize the learning of novel classes.
- Score: 16.461242381109276
- License: http://creativecommons.org/licenses/by-nc-nd/4.0/
- Abstract: We tackle the problem of novel class discovery, which aims to learn novel
classes without supervision based on labeled data from known classes. A key
challenge lies in transferring the knowledge in the known-class data to the
learning of novel classes. Previous methods mainly focus on building a shared
representation space for knowledge transfer and often ignore modeling class
relations. To address this, we introduce a class relation representation for
the novel classes based on the predicted class distribution of a model trained
on known classes. Empirically, we find that such class relation becomes less
informative during typical discovery training. To prevent such information
loss, we propose a novel knowledge distillation framework, which utilizes our
class-relation representation to regularize the learning of novel classes. In
addition, to enable a flexible knowledge distillation scheme for each data
point in novel classes, we develop a learnable weighting function for the
regularization, which adaptively promotes knowledge transfer based on the
semantic similarity between the novel and known classes. To validate the
effectiveness and generalization of our method, we conduct extensive
experiments on multiple benchmarks, including CIFAR100, Stanford Cars, CUB, and
FGVC-Aircraft datasets. Our results demonstrate that the proposed method
outperforms the previous state-of-the-art methods by a significant margin on
almost all benchmarks. Code is available at
\href{https://github.com/kleinzcy/Cr-KD-NCD}{here}.
Related papers
- Self-Cooperation Knowledge Distillation for Novel Class Discovery [8.984031974257274]
Novel Class Discovery (NCD) aims to discover unknown and novel classes in an unlabeled set by leveraging knowledge already learned about known classes.
We propose a Self-Cooperation Knowledge Distillation (SCKD) method to utilize each training sample (whether known or novel, labeled or unlabeled) for both review and discovery.
arXiv Detail & Related papers (2024-07-02T03:49:48Z) - A Unified and General Framework for Continual Learning [58.72671755989431]
Continual Learning (CL) focuses on learning from dynamic and changing data distributions while retaining previously acquired knowledge.
Various methods have been developed to address the challenge of catastrophic forgetting, including regularization-based, Bayesian-based, and memory-replay-based techniques.
This research aims to bridge this gap by introducing a comprehensive and overarching framework that encompasses and reconciles these existing methodologies.
arXiv Detail & Related papers (2024-03-20T02:21:44Z) - Continual Learning with Pre-Trained Models: A Survey [61.97613090666247]
Continual Learning aims to overcome the catastrophic forgetting of former knowledge when learning new ones.
This paper presents a comprehensive survey of the latest advancements in PTM-based CL.
arXiv Detail & Related papers (2024-01-29T18:27:52Z) - Novel Class Discovery for Long-tailed Recognition [17.464885951359047]
We consider a more realistic setting for novel class discovery where the distributions of novel and known classes are long-tailed.
To tackle this problem, we propose an adaptive self-labeling strategy based on an equiangular prototype representation of classes.
Our method infers high-quality pseudo-labels for the novel classes by solving a relaxed optimal transport problem.
arXiv Detail & Related papers (2023-08-06T02:15:19Z) - Large-scale Pre-trained Models are Surprisingly Strong in Incremental Novel Class Discovery [76.63807209414789]
We challenge the status quo in class-iNCD and propose a learning paradigm where class discovery occurs continuously and truly unsupervisedly.
We propose simple baselines, composed of a frozen PTM backbone and a learnable linear classifier, that are not only simple to implement but also resilient under longer learning scenarios.
arXiv Detail & Related papers (2023-03-28T13:47:16Z) - Novel Class Discovery without Forgetting [72.52222295216062]
We identify and formulate a new, pragmatic problem setting of NCDwF: Novel Class Discovery without Forgetting.
We propose a machine learning model to incrementally discover novel categories of instances from unlabeled data.
We introduce experimental protocols based on CIFAR-10, CIFAR-100 and ImageNet-1000 to measure the trade-off between knowledge retention and novel class discovery.
arXiv Detail & Related papers (2022-07-21T17:54:36Z) - Class-incremental Novel Class Discovery [76.35226130521758]
We study the new task of class-incremental Novel Class Discovery (class-iNCD)
We propose a novel approach for class-iNCD which prevents forgetting of past information about the base classes.
Our experiments, conducted on three common benchmarks, demonstrate that our method significantly outperforms state-of-the-art approaches.
arXiv Detail & Related papers (2022-07-18T13:49:27Z) - Mutual Information-guided Knowledge Transfer for Novel Class Discovery [23.772336970389834]
We propose a principle and general method to transfer semantic knowledge between seen and unseen classes.
Our results show that the proposed method outperforms previous SOTA by a significant margin on several benchmarks.
arXiv Detail & Related papers (2022-06-24T03:52:25Z) - Learning to Generate Novel Classes for Deep Metric Learning [24.048915378172012]
We introduce a new data augmentation approach that synthesizes novel classes and their embedding vectors.
We implement this idea by learning and exploiting a conditional generative model, which, given a class label and a noise, produces a random embedding vector of the class.
Our proposed generator allows the loss to use richer class relations by augmenting realistic and diverse classes, resulting in better generalization to unseen samples.
arXiv Detail & Related papers (2022-01-04T06:55:19Z) - Long-tail Recognition via Compositional Knowledge Transfer [60.03764547406601]
We introduce a novel strategy for long-tail recognition that addresses the tail classes' few-shot problem.
Our objective is to transfer knowledge acquired from information-rich common classes to semantically similar, and yet data-hungry, rare classes.
Experiments show that our approach can achieve significant performance boosts on rare classes while maintaining robust common class performance.
arXiv Detail & Related papers (2021-12-13T15:48:59Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.