The Triad of Failure Modes and a Possible Way Out
- URL: http://arxiv.org/abs/2309.15420v1
- Date: Wed, 27 Sep 2023 05:54:14 GMT
- Title: The Triad of Failure Modes and a Possible Way Out
- Authors: Emanuele Sansone
- Abstract summary: We present a novel objective function for cluster-based self-supervised learning (SSL) that is designed to circumvent the triad of failure modes.
This objective consists of three key components: (i) A generative term that penalizes representation collapse, (ii) a term that promotes invariance to data augmentations, and (ii) a uniformity term that penalizes cluster collapse.
- Score: 7.977229957867868
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: We present a novel objective function for cluster-based self-supervised
learning (SSL) that is designed to circumvent the triad of failure modes,
namely representation collapse, cluster collapse, and the problem of invariance
to permutations of cluster assignments. This objective consists of three key
components: (i) A generative term that penalizes representation collapse, (ii)
a term that promotes invariance to data augmentations, thereby addressing the
issue of label permutations and (ii) a uniformity term that penalizes cluster
collapse. Additionally, our proposed objective possesses two notable
advantages. Firstly, it can be interpreted from a Bayesian perspective as a
lower bound on the data log-likelihood. Secondly, it enables the training of a
standard backbone architecture without the need for asymmetric elements like
stop gradients, momentum encoders, or specialized clustering layers. Due to its
simplicity and theoretical foundation, our proposed objective is well-suited
for optimization. Experiments on both toy and real world data demonstrate its
effectiveness
Related papers
- A Contrastive Variational Graph Auto-Encoder for Node Clustering [10.52321770126932]
State-of-the-art clustering methods have numerous challenges.
Existing VGAEs do not account for the discrepancy between the inference and generative models.
Our solution has two mechanisms to control the trade-off between Feature Randomness and Feature Drift.
arXiv Detail & Related papers (2023-12-28T05:07:57Z) - Neural Collapse Terminus: A Unified Solution for Class Incremental
Learning and Its Variants [166.916517335816]
In this paper, we offer a unified solution to the misalignment dilemma in the three tasks.
We propose neural collapse terminus that is a fixed structure with the maximal equiangular inter-class separation for the whole label space.
Our method holds the neural collapse optimality in an incremental fashion regardless of data imbalance or data scarcity.
arXiv Detail & Related papers (2023-08-03T13:09:59Z) - Data Augmentation-free Unsupervised Learning for 3D Point Cloud
Understanding [61.30276576646909]
We propose an augmentation-free unsupervised approach for point clouds to learn transferable point-level features via soft clustering, named SoftClu.
We exploit the affiliation of points to their clusters as a proxy to enable self-training through a pseudo-label prediction task.
arXiv Detail & Related papers (2022-10-06T10:18:16Z) - Unsupervised Learning on 3D Point Clouds by Clustering and Contrasting [11.64827192421785]
unsupervised representation learning is a promising direction to auto-extract features without human intervention.
This paper proposes a general unsupervised approach, named textbfConClu, to perform the learning of point-wise and global features.
arXiv Detail & Related papers (2022-02-05T12:54:17Z) - Guided Point Contrastive Learning for Semi-supervised Point Cloud
Semantic Segmentation [90.2445084743881]
We present a method for semi-supervised point cloud semantic segmentation to adopt unlabeled point clouds in training to boost the model performance.
Inspired by the recent contrastive loss in self-supervised tasks, we propose the guided point contrastive loss to enhance the feature representation and model generalization ability.
arXiv Detail & Related papers (2021-10-15T16:38:54Z) - Clustering by Maximizing Mutual Information Across Views [62.21716612888669]
We propose a novel framework for image clustering that incorporates joint representation learning and clustering.
Our method significantly outperforms state-of-the-art single-stage clustering methods across a variety of image datasets.
arXiv Detail & Related papers (2021-07-24T15:36:49Z) - You Never Cluster Alone [150.94921340034688]
We extend the mainstream contrastive learning paradigm to a cluster-level scheme, where all the data subjected to the same cluster contribute to a unified representation.
We define a set of categorical variables as clustering assignment confidence, which links the instance-level learning track with the cluster-level one.
By reparametrizing the assignment variables, TCC is trained end-to-end, requiring no alternating steps.
arXiv Detail & Related papers (2021-06-03T14:59:59Z) - Provably End-to-end Label-Noise Learning without Anchor Points [118.97592870124937]
We propose an end-to-end framework for solving label-noise learning without anchor points.
Our proposed framework can identify the transition matrix if the clean class-posterior probabilities are sufficiently scattered.
arXiv Detail & Related papers (2021-02-04T03:59:37Z) - Canonical Capsules: Unsupervised Capsules in Canonical Pose [31.529908150084626]
We propose an unsupervised capsule architecture for 3D point clouds.
We compute capsule decompositions of objects through permutation-equivariant attention, and self-supervise the process by training with pairs of randomly rotated objects.
By learning an object-centric representation in an unsupervised manner, our method outperforms the state-of-the-art on 3D point cloud reconstruction, registration, and unsupervised classification.
arXiv Detail & Related papers (2020-12-08T20:13:28Z) - Learning Embeddings for Image Clustering: An Empirical Study of Triplet
Loss Approaches [10.42820615166362]
We evaluate two different image clustering objectives, k-means clustering and correlation clustering, in the context of Triplet Loss induced feature space embeddings.
We train a convolutional neural network to learn discriminative features by optimizing two popular versions of the Triplet Loss.
We propose a new, simple Triplet Loss formulation, which shows desirable properties with respect to formal clustering objectives and outperforms the existing methods.
arXiv Detail & Related papers (2020-07-06T23:38:14Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.