Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines

• URL: http://arxiv.org/abs/2107.06013v2
• Date: Thu, 21 Oct 2021 07:39:00 GMT
• Title: Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines
• Authors: Cl\'ement Roussel, Simona Cocco, R\'emi Monasson
• Abstract summary: We study the performance of Alternating Gibbs Sampling (AGS) on several analytically tractable models. We show that standard AGS is not more efficient than classical Metropolis-Hastings (MH) sampling of the effective energy landscape. We illustrate our findings on three datasets: Bars and Stripes and MNIST, well known in machine learning, and the so-called Lattice Proteins.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Restricted Boltzmann Machines (RBM) are bi-layer neural networks used for the unsupervised learning of model distributions from data. The bipartite architecture of RBM naturally defines an elegant sampling procedure, called Alternating Gibbs Sampling (AGS), where the configurations of the latent-variable layer are sampled conditional to the data-variable layer, and vice versa. We study here the performance of AGS on several analytically tractable models borrowed from statistical mechanics. We show that standard AGS is not more efficient than classical Metropolis-Hastings (MH) sampling of the effective energy landscape defined on the data layer. However, RBM can identify meaningful representations of training data in their latent space. Furthermore, using these representations and combining Gibbs sampling with the MH algorithm in the latent space can enhance the sampling performance of the RBM when the hidden units encode weakly dependent features of the data. We illustrate our findings on three datasets: Bars and Stripes and MNIST, well known in machine learning, and the so-called Lattice Proteins, introduced in theoretical biology to study the sequence-to-structure mapping in proteins.

Related papers

• GFM4MPM: Towards Geospatial Foundation Models for Mineral Prospectivity Mapping [2.7998963147546148]
  We propose a self-supervised approach to train a backbone neural network in a self-supervised manner using unlabeled geospatial data alone. Our results demonstrate that self-supervision promotes robustness in learned features, improving prospectivity predictions. We leverage explainable artificial intelligence techniques to demonstrate that individual predictions can be interpreted from a geological perspective.
  arXiv  Detail & Related papers  (2024-06-18T16:24:28Z)
• Synthetic location trajectory generation using categorical diffusion models [50.809683239937584]
  Diffusion models (DPMs) have rapidly evolved to be one of the predominant generative models for the simulation of synthetic data. We propose using DPMs for the generation of synthetic individual location trajectories (ILTs) which are sequences of variables representing physical locations visited by individuals.
  arXiv  Detail & Related papers  (2024-02-19T15:57:39Z)
• cDVGAN: One Flexible Model for Multi-class Gravitational Wave Signal and Glitch Generation [0.7853804618032806]
  We present a novel conditional model in the Generative Adrial Network framework for simulating multiple classes of time-domain observations. Our proposed cDVGAN outperforms 4 different baseline GAN models in replicating the features of the three classes. Our experiments show that training convolutional neural networks with our cDVGAN-generated data improves the detection of samples embedded in detector noise.
  arXiv  Detail & Related papers  (2024-01-29T17:59:26Z)
• LD-GAN: Low-Dimensional Generative Adversarial Network for Spectral Image Generation with Variance Regularization [72.4394510913927]
  Deep learning methods are state-of-the-art for spectral image (SI) computational tasks. GANs enable diverse augmentation by learning and sampling from the data distribution. GAN-based SI generation is challenging since the high-dimensionality nature of this kind of data hinders the convergence of the GAN training yielding to suboptimal generation. We propose a statistical regularization to control the low-dimensional representation variance for the autoencoder training and to achieve high diversity of samples generated with the GAN.
  arXiv  Detail & Related papers  (2023-04-29T00:25:02Z)
• Automated classification of pre-defined movement patterns: A comparison between GNSS and UWB technology [55.41644538483948]
  Real-time location systems (RTLS) allow for collecting data from human movement patterns. The current study aims to design and evaluate an automated framework to classify human movement patterns in small areas.
  arXiv  Detail & Related papers  (2023-03-10T14:46:42Z)
• ContrasInver: Ultra-Sparse Label Semi-supervised Regression for Multi-dimensional Seismic Inversion [7.356328937024184]
  ContrasInver is a method that achieves seismic inversion using as few as two or three well logs. In experiments, ContrasInver achieved state-of-the-art performance in the synthetic data SEAM I. It's the first data-driven approach yielding reliable results on the Netherlands F3 and Delft, using only three and two well logs respectively.
  arXiv  Detail & Related papers  (2023-02-13T15:19:51Z)
• The emergence of a concept in shallow neural networks [0.0]
  We consider restricted Boltzmann machine (RBMs) trained over an unstructured dataset made of blurred copies of definite but unavailable archetypes'' We show that there exists a critical sample size beyond which the RBM can learn archetypes.
  arXiv  Detail & Related papers  (2021-09-01T15:56:38Z)
• Diverse Sample Generation: Pushing the Limit of Data-free Quantization [85.95032037447454]
  This paper presents a generic Diverse Sample Generation scheme for the generative data-free post-training quantization and quantization-aware training. For large-scale image classification tasks, our DSG can consistently outperform existing data-free quantization methods.
  arXiv  Detail & Related papers  (2021-09-01T07:06:44Z)
• ADAVI: Automatic Dual Amortized Variational Inference Applied To Pyramidal Bayesian Models [0.0]
  We develop a novel methodology that automatically produces a variatonal family dual to a target parameter. We demonstrate the capability of our method on simulated data, as well as a challenging high-dimensional brain parcellation experiment.
  arXiv  Detail & Related papers  (2021-06-23T09:09:01Z)
• G-MIND: An End-to-End Multimodal Imaging-Genetics Framework for Biomarker Identification and Disease Classification [49.53651166356737]
  We propose a novel deep neural network architecture to integrate imaging and genetics data, as guided by diagnosis, that provides interpretable biomarkers. We have evaluated our model on a population study of schizophrenia that includes two functional MRI (fMRI) paradigms and Single Nucleotide Polymorphism (SNP) data.
  arXiv  Detail & Related papers  (2021-01-27T19:28:04Z)
• Understanding Self-supervised Learning with Dual Deep Networks [74.92916579635336]
  We propose a novel framework to understand contrastive self-supervised learning (SSL) methods that employ dual pairs of deep ReLU networks. We prove that in each SGD update of SimCLR with various loss functions, the weights at each layer are updated by a emphcovariance operator. To further study what role the covariance operator plays and which features are learned in such a process, we model data generation and augmentation processes through a emphhierarchical latent tree model (HLTM)
  arXiv  Detail & Related papers  (2020-10-01T17:51:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.