Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data

• URL: http://arxiv.org/abs/2104.12673v2
• Date: Tue, 27 Apr 2021 09:00:44 GMT
• Title: Joint Representation Learning and Novel Category Discovery on Single- and Multi-modal Data
• Authors: Xuhui Jia and Kai Han and Yukun Zhu and Bradley Green
• Abstract summary: We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. We employ Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet.
• Score: 16.138075558585516
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper studies the problem of novel category discovery on single- and multi-modal data with labels from different but relevant categories. We present a generic, end-to-end framework to jointly learn a reliable representation and assign clusters to unlabelled data. To avoid over-fitting the learnt embedding to labelled data, we take inspiration from self-supervised representation learning by noise-contrastive estimation and extend it to jointly handle labelled and unlabelled data. In particular, we propose using category discrimination on labelled data and cross-modal discrimination on multi-modal data to augment instance discrimination used in conventional contrastive learning approaches. We further employ Winner-Take-All (WTA) hashing algorithm on the shared representation space to generate pairwise pseudo labels for unlabelled data to better predict cluster assignments. We thoroughly evaluate our framework on large-scale multi-modal video benchmarks Kinetics-400 and VGG-Sound, and image benchmarks CIFAR10, CIFAR100 and ImageNet, obtaining state-of-the-art results.

Related papers

• Cross-Modality Clustering-based Self-Labeling for Multimodal Data Classification [2.666791490663749]
  Cross-Modality Clustering-based Self-Labeling ( CMCSL) CMCSL groups instances belonging to each modality in the deep feature space and then propagates known labels within the resulting clusters. Experimental evaluation conducted on 20 datasets derived from the MM-IMDb dataset.
  arXiv  Detail & Related papers  (2024-08-05T15:43:56Z)
• Self Supervised Correlation-based Permutations for Multi-View Clustering [7.972599673048582]
  We propose an end-to-end deep learning-based MVC framework for general data. Our approach involves learning meaningful fused data representations with a novel permutation-based canonical correlation objective. We demonstrate the effectiveness of our model using ten MVC benchmark datasets.
  arXiv  Detail & Related papers  (2024-02-26T08:08:30Z)
• Memory Consistency Guided Divide-and-Conquer Learning for Generalized Category Discovery [56.172872410834664]
  Generalized category discovery (GCD) aims at addressing a more realistic and challenging setting of semi-supervised learning. We propose a Memory Consistency guided Divide-and-conquer Learning framework (MCDL) Our method outperforms state-of-the-art models by a large margin on both seen and unseen classes of the generic image recognition.
  arXiv  Detail & Related papers  (2024-01-24T09:39:45Z)
• Generalized Category Discovery with Clustering Assignment Consistency [56.92546133591019]
  Generalized category discovery (GCD) is a recently proposed open-world task. We propose a co-training-based framework that encourages clustering consistency. Our method achieves state-of-the-art performance on three generic benchmarks and three fine-grained visual recognition datasets.
  arXiv  Detail & Related papers  (2023-10-30T00:32:47Z)
• Tackling Diverse Minorities in Imbalanced Classification [80.78227787608714]
  Imbalanced datasets are commonly observed in various real-world applications, presenting significant challenges in training classifiers. We propose generating synthetic samples iteratively by mixing data samples from both minority and majority classes. We demonstrate the effectiveness of our proposed framework through extensive experiments conducted on seven publicly available benchmark datasets.
  arXiv  Detail & Related papers  (2023-08-28T18:48:34Z)
• Learning Semantic Segmentation from Multiple Datasets with Label Shifts [101.24334184653355]
  This paper proposes UniSeg, an effective approach to automatically train models across multiple datasets with differing label spaces. Specifically, we propose two losses that account for conflicting and co-occurring labels to achieve better generalization performance in unseen domains.
  arXiv  Detail & Related papers  (2022-02-28T18:55:19Z)
• Novel Visual Category Discovery with Dual Ranking Statistics and Mutual Knowledge Distillation [16.357091285395285]
  We tackle the problem of grouping unlabelled images from new classes into different semantic partitions. This is a more realistic and challenging setting than conventional semi-supervised learning. We propose a two-branch learning framework for this problem, with one branch focusing on local part-level information and the other branch focusing on overall characteristics.
  arXiv  Detail & Related papers  (2021-07-07T17:14:40Z)
• Neighborhood Contrastive Learning for Novel Class Discovery [79.14767688903028]
  We build a new framework, named Neighborhood Contrastive Learning, to learn discriminative representations that are important to clustering performance. We experimentally demonstrate that these two ingredients significantly contribute to clustering performance and lead our model to outperform state-of-the-art methods by a large margin.
  arXiv  Detail & Related papers  (2021-06-20T17:34:55Z)
• Automatically Discovering and Learning New Visual Categories with Ranking Statistics [145.89790963544314]
  We tackle the problem of discovering novel classes in an image collection given labelled examples of other classes. We learn a general-purpose clustering model and use the latter to identify the new classes in the unlabelled data. We evaluate our approach on standard classification benchmarks and outperform current methods for novel category discovery by a significant margin.
  arXiv  Detail & Related papers  (2020-02-13T18:53:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.