A Probabilistic Approach to Self-Supervised Learning using Cyclical Stochastic Gradient MCMC

• URL: http://arxiv.org/abs/2308.01271v1
• Date: Wed, 2 Aug 2023 16:52:56 GMT
• Title: A Probabilistic Approach to Self-Supervised Learning using Cyclical Stochastic Gradient MCMC
• Authors: Masoumeh Javanbakhat, Christoph Lippert
• Abstract summary: We present a practical self-supervised learning method with Cyclical Gradient Hamiltonian Monte Carlo (cSGHMC) Within this framework, we place a prior over the parameters of a self-supervised learning model and use cSGHMC to approximate the high dimensional and multimodal posterior distribution over the embeddings. We provide experimental results on multiple classification tasks on four challenging datasets.
• Score: 8.027994148508844
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper we present a practical Bayesian self-supervised learning method with Cyclical Stochastic Gradient Hamiltonian Monte Carlo (cSGHMC). Within this framework, we place a prior over the parameters of a self-supervised learning model and use cSGHMC to approximate the high dimensional and multimodal posterior distribution over the embeddings. By exploring an expressive posterior over the embeddings, Bayesian self-supervised learning produces interpretable and diverse representations. Marginalizing over these representations yields a significant gain in performance, calibration and out-of-distribution detection on a variety of downstream classification tasks. We provide experimental results on multiple classification tasks on four challenging datasets. Moreover, we demonstrate the effectiveness of the proposed method in out-of-distribution detection using the SVHN and CIFAR-10 datasets.

Related papers

• On Discriminative Probabilistic Modeling for Self-Supervised Representation Learning [85.75164588939185]
  We study the discriminative probabilistic modeling problem on a continuous domain for (multimodal) self-supervised representation learning. We conduct generalization error analysis to reveal the limitation of current InfoNCE-based contrastive loss for self-supervised representation learning.
  arXiv  Detail & Related papers  (2024-10-11T18:02:46Z)
• Hierarchical Visual Categories Modeling: A Joint Representation Learning and Density Estimation Framework for Out-of-Distribution Detection [28.442470704073767]
  This paper proposes a novel hierarchical visual category modeling scheme to separate out-of-distribution data from in-distribution data. We conduct experiments on seven popular benchmarks, including CIFAR, iNaturalist, SUN, Places, Textures, ImageNet-O, and OpenImage-O. Our visual representation has a competitive performance when compared with features learned by classical methods.
  arXiv  Detail & Related papers  (2024-08-28T07:05:46Z)
• Learning to Explore for Stochastic Gradient MCMC [15.286308920219446]
  We propose a meta-learning strategy to build glssgmcmc which can efficiently explore the multi-modal target distributions. Our algorithm allows the learned SGMCMC to quickly explore the high-density region of the posterior landscape.
  arXiv  Detail & Related papers  (2024-08-17T08:36:42Z)
• Convolutional autoencoder-based multimodal one-class classification [80.52334952912808]
  One-class classification refers to approaches of learning using data from a single class only. We propose a deep learning one-class classification method suitable for multimodal data.
  arXiv  Detail & Related papers  (2023-09-25T12:31:18Z)
• Ensemble Modeling for Multimodal Visual Action Recognition [50.38638300332429]
  We propose an ensemble modeling approach for multimodal action recognition. We independently train individual modality models using a variant of focal loss tailored to handle the long-tailed distribution of the MECCANO [21] dataset.
  arXiv  Detail & Related papers  (2023-08-10T08:43:20Z)
• Manifold Contrastive Learning with Variational Lie Group Operators [5.0741409008225755]
  We propose a contrastive learning approach that directly models the latent manifold using Lie group operators parameterized by coefficients with a sparsity-promoting prior. A variational distribution over these coefficients provides a generative model of the manifold, with samples which provide feature augmentations applicable both during contrastive training and downstream tasks.
  arXiv  Detail & Related papers  (2023-06-23T15:07:01Z)
• Learning disentangled representations for explainable chest X-ray classification using Dirichlet VAEs [68.73427163074015]
  This study explores the use of the Dirichlet Variational Autoencoder (DirVAE) for learning disentangled latent representations of chest X-ray (CXR) images. The predictive capacity of multi-modal latent representations learned by DirVAE models is investigated through implementation of an auxiliary multi-label classification task.
  arXiv  Detail & Related papers  (2023-02-06T18:10:08Z)
• MCDAL: Maximum Classifier Discrepancy for Active Learning [74.73133545019877]
  Recent state-of-the-art active learning methods have mostly leveraged Generative Adversarial Networks (GAN) for sample acquisition. We propose in this paper a novel active learning framework that we call Maximum Discrepancy for Active Learning (MCDAL) In particular, we utilize two auxiliary classification layers that learn tighter decision boundaries by maximizing the discrepancies among them.
  arXiv  Detail & Related papers  (2021-07-23T06:57:08Z)
• Improving Deep Learning Sound Events Classifiers using Gram Matrix Feature-wise Correlations [1.2891210250935146]
  In our method, we analyse all the activations of a generic CNN in order to produce feature representations using Gram Matrices. The proposed approach can be applied to any CNN and our experimental evaluation of four different architectures on two datasets demonstrated that our method consistently improves the baseline models.
  arXiv  Detail & Related papers  (2021-02-23T16:08:02Z)
• Trusted Multi-View Classification [76.73585034192894]
  We propose a novel multi-view classification method, termed trusted multi-view classification. It provides a new paradigm for multi-view learning by dynamically integrating different views at an evidence level. The proposed algorithm jointly utilizes multiple views to promote both classification reliability and robustness.
  arXiv  Detail & Related papers  (2021-02-03T13:30:26Z)
• Ask-n-Learn: Active Learning via Reliable Gradient Representations for Image Classification [29.43017692274488]
  Deep predictive models rely on human supervision in the form of labeled training data. We propose Ask-n-Learn, an active learning approach based on gradient embeddings obtained using the pesudo-labels estimated in each of the algorithm.
  arXiv  Detail & Related papers  (2020-09-30T05:19:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.