Domain-Agnostic Clustering with Self-Distillation

• URL: http://arxiv.org/abs/2111.12170v1
• Date: Tue, 23 Nov 2021 21:56:54 GMT
• Title: Domain-Agnostic Clustering with Self-Distillation
• Authors: Mohammed Adnan, Yani A. Ioannou, Chuan-Yung Tsai, Graham W. Taylor
• Abstract summary: We propose a new self-distillation based algorithm for domain-agnostic clustering. We empirically demonstrate that knowledge distillation can improve unsupervised representation learning. Preliminary experiments also suggest that self-distillation improves the convergence of DeepCluster-v2.
• Score: 21.58831206727797
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in self-supervised learning have reduced the gap between supervised and unsupervised representation learning. However, most self-supervised and deep clustering techniques rely heavily on data augmentation, rendering them ineffective for many learning tasks where insufficient domain knowledge exists for performing augmentation. We propose a new self-distillation based algorithm for domain-agnostic clustering. Our method builds upon the existing deep clustering frameworks and requires no separate student model. The proposed method outperforms existing domain agnostic (augmentation-free) algorithms on CIFAR-10. We empirically demonstrate that knowledge distillation can improve unsupervised representation learning by extracting richer `dark knowledge' from the model than using predicted labels alone. Preliminary experiments also suggest that self-distillation improves the convergence of DeepCluster-v2.

Related papers

• Unsupervised Cognition [2.5069344340760713]
  We propose a primitive-based, unsupervised learning approach for decision-making inspired by a novel cognition framework.<n>This representation-centric approach models the input space constructively as a distributed hierarchical structure in an input-agnostic way.
  arXiv  Detail & Related papers  (2024-09-27T10:50:49Z)
• Counterfactual Explanations for Clustering Models [11.40145394568897]
  Clustering algorithms rely on complex optimisation processes that may be difficult to comprehend. We propose a new, model-agnostic technique for explaining clustering algorithms with counterfactual statements.
  arXiv  Detail & Related papers  (2024-09-19T10:05:58Z)
• GCC: Generative Calibration Clustering [55.44944397168619]
  We propose a novel Generative Clustering (GCC) method to incorporate feature learning and augmentation into clustering procedure. First, we develop a discrimirative feature alignment mechanism to discover intrinsic relationship across real and generated samples. Second, we design a self-supervised metric learning to generate more reliable cluster assignment.
  arXiv  Detail & Related papers  (2024-04-14T01:51:11Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Adversarial Auto-Augment with Label Preservation: A Representation Learning Principle Guided Approach [95.74102207187545]
  We show that a prior-free autonomous data augmentation's objective can be derived from a representation learning principle. We then propose a practical surrogate to the objective that can be efficiently optimized and integrated seamlessly into existing methods.
  arXiv  Detail & Related papers  (2022-11-02T02:02:51Z)
• A Deep Dive into Deep Cluster [0.2578242050187029]
  DeepCluster is a simple and scalable unsupervised pretraining of visual representations. We show that DeepCluster convergence and performance depend on the interplay between the quality of the randomly filters of the convolutional layer and the selected number of clusters.
  arXiv  Detail & Related papers  (2022-07-24T22:55:09Z)
• Chaos is a Ladder: A New Theoretical Understanding of Contrastive Learning via Augmentation Overlap [64.60460828425502]
  We propose a new guarantee on the downstream performance of contrastive learning. Our new theory hinges on the insight that the support of different intra-class samples will become more overlapped under aggressive data augmentations. We propose an unsupervised model selection metric ARC that aligns well with downstream accuracy.
  arXiv  Detail & Related papers  (2022-03-25T05:36:26Z)
• On Generalizing Beyond Domains in Cross-Domain Continual Learning [91.56748415975683]
  Deep neural networks often suffer from catastrophic forgetting of previously learned knowledge after learning a new task. Our proposed approach learns new tasks under domain shift with accuracy boosts up to 10% on challenging datasets such as DomainNet and OfficeHome.
  arXiv  Detail & Related papers  (2022-03-08T09:57:48Z)
• Deep Attention-guided Graph Clustering with Dual Self-supervision [49.040136530379094]
  We propose a novel method, namely deep attention-guided graph clustering with dual self-supervision (DAGC) We develop a dual self-supervision solution consisting of a soft self-supervision strategy with a triplet Kullback-Leibler divergence loss and a hard self-supervision strategy with a pseudo supervision loss. Our method consistently outperforms state-of-the-art methods on six benchmark datasets.
  arXiv  Detail & Related papers  (2021-11-10T06:53:03Z)
• Contrastive Neural Processes for Self-Supervised Learning [1.8059331230167266]
  We propose a novel self-supervised learning framework that combines contrastive learning with neural processes. It relies on recent advances in neural processes to perform time series forecasting. Unlike previous self-supervised methods, our augmentation pipeline is task-agnostic, enabling our method to perform well across various applications.
  arXiv  Detail & Related papers  (2021-10-24T21:01:27Z)
• Distill on the Go: Online knowledge distillation in self-supervised learning [1.1470070927586016]
  Recent works have shown that wider and deeper models benefit more from self-supervised learning than smaller models. We propose Distill-on-the-Go (DoGo), a self-supervised learning paradigm using single-stage online knowledge distillation. Our results show significant performance gain in the presence of noisy and limited labels.
  arXiv  Detail & Related papers  (2021-04-20T09:59:23Z)
• Consensus Clustering With Unsupervised Representation Learning [4.164845768197489]
  We study the clustering ability of Bootstrap Your Own Latent (BYOL) and observe that features learnt using BYOL may not be optimal for clustering. We propose a novel consensus clustering based loss function, and train BYOL with the proposed loss in an end-to-end way that improves the clustering ability and outperforms similar clustering based methods.
  arXiv  Detail & Related papers  (2020-10-03T01:16:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.