Clustering Properties of Self-Supervised Learning

• URL: http://arxiv.org/abs/2501.18452v1
• Date: Thu, 30 Jan 2025 16:05:35 GMT
• Title: Clustering Properties of Self-Supervised Learning
• Authors: Xi Weng, Jianing An, Xudong Ma, Binhang Qi, Jie Luo, Xi Yang, Jin Song Dong, Lei Huang,
• Abstract summary: Self-supervised learning (SSL) methods have proven remarkably effective at capturing semantically rich representations with strong clustering properties. We propose a novel positive-feedback SSL method, termed Representation Soft Assignment (ReSA), which leverages the model's clustering properties to promote learning in a self-guided manner.
• Score: 14.756786256090704
• License:
• Abstract: Self-supervised learning (SSL) methods via joint embedding architectures have proven remarkably effective at capturing semantically rich representations with strong clustering properties, magically in the absence of label supervision. Despite this, few of them have explored leveraging these untapped properties to improve themselves. In this paper, we provide an evidence through various metrics that the encoder's output $encoding$ exhibits superior and more stable clustering properties compared to other components. Building on this insight, we propose a novel positive-feedback SSL method, termed Representation Soft Assignment (ReSA), which leverages the model's clustering properties to promote learning in a self-guided manner. Extensive experiments on standard SSL benchmarks reveal that models pretrained with ReSA outperform other state-of-the-art SSL methods by a significant margin. Finally, we analyze how ReSA facilitates better clustering properties, demonstrating that it effectively enhances clustering performance at both fine-grained and coarse-grained levels, shaping representations that are inherently more structured and semantically meaningful.

Related papers

• GCC: Generative Calibration Clustering [55.44944397168619]
  We propose a novel Generative Clustering (GCC) method to incorporate feature learning and augmentation into clustering procedure. First, we develop a discrimirative feature alignment mechanism to discover intrinsic relationship across real and generated samples. Second, we design a self-supervised metric learning to generate more reliable cluster assignment.
  arXiv  Detail & Related papers  (2024-04-14T01:51:11Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Reverse Engineering Self-Supervised Learning [17.720366509919167]
  Self-supervised learning (SSL) is a powerful tool in machine learning. This paper presents an in-depth empirical analysis of SSL-trained representations.
  arXiv  Detail & Related papers  (2023-05-24T23:15:28Z)
• Using Representation Expressiveness and Learnability to Evaluate Self-Supervised Learning Methods [61.49061000562676]
  We introduce Cluster Learnability (CL) to assess learnability. CL is measured in terms of the performance of a KNN trained to predict labels obtained by clustering the representations with K-means. We find that CL better correlates with in-distribution model performance than other competing recent evaluation schemes.
  arXiv  Detail & Related papers  (2022-06-02T19:05:13Z)
• Weak Augmentation Guided Relational Self-Supervised Learning [80.0680103295137]
  We introduce a novel relational self-supervised learning (ReSSL) framework that learns representations by modeling the relationship between different instances. Our proposed method employs sharpened distribution of pairwise similarities among different instances as textitrelation metric. Experimental results show that our proposed ReSSL substantially outperforms the state-of-the-art methods across different network architectures.
  arXiv  Detail & Related papers  (2022-03-16T16:14:19Z)
• Structure-Aware Feature Generation for Zero-Shot Learning [108.76968151682621]
  We introduce a novel structure-aware feature generation scheme, termed as SA-GAN, to account for the topological structure in learning both the latent space and the generative networks. Our method significantly enhances the generalization capability on unseen-classes and consequently improve the classification performance.
  arXiv  Detail & Related papers  (2021-08-16T11:52:08Z)
• ReSSL: Relational Self-Supervised Learning with Weak Augmentation [68.47096022526927]
  Self-supervised learning has achieved great success in learning visual representations without data annotations. We introduce a novel relational SSL paradigm that learns representations by modeling the relationship between different instances. Our proposed ReSSL significantly outperforms the previous state-of-the-art algorithms in terms of both performance and training efficiency.
  arXiv  Detail & Related papers  (2021-07-20T06:53:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.