Gaussian-Bernoulli RBMs Without Tears

• URL: http://arxiv.org/abs/2210.10318v1
• Date: Wed, 19 Oct 2022 06:22:55 GMT
• Title: Gaussian-Bernoulli RBMs Without Tears
• Authors: Renjie Liao, Simon Kornblith, Mengye Ren, David J. Fleet, Geoffrey Hinton
• Abstract summary: We propose a novel Gibbs-Langevin sampling algorithm that outperforms existing methods like Gibbs sampling. We propose a modified contrastive divergence (CD) algorithm so that one can generate images with GRBMs starting from noise.
• Score: 113.62579223055958
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We revisit the challenging problem of training Gaussian-Bernoulli restricted Boltzmann machines (GRBMs), introducing two innovations. We propose a novel Gibbs-Langevin sampling algorithm that outperforms existing methods like Gibbs sampling. We propose a modified contrastive divergence (CD) algorithm so that one can generate images with GRBMs starting from noise. This enables direct comparison of GRBMs with deep generative models, improving evaluation protocols in the RBM literature. Moreover, we show that modified CD and gradient clipping are enough to robustly train GRBMs with large learning rates, thus removing the necessity of various tricks in the literature. Experiments on Gaussian Mixtures, MNIST, FashionMNIST, and CelebA show GRBMs can generate good samples, despite their single-hidden-layer architecture. Our code is released at: \url{https://github.com/lrjconan/GRBM}.

Related papers

• GaussianDreamerPro: Text to Manipulable 3D Gaussians with Highly Enhanced Quality [99.63429416013713]
  3D-GS has achieved great success in reconstructing and rendering real-world scenes. To transfer the high rendering quality to generation tasks, a series of research works attempt to generate 3D-Gaussian assets from text. We propose a novel framework named GaussianDreamerPro to enhance the generation quality.
  arXiv  Detail & Related papers  (2024-06-26T16:12:09Z)
• 3D Gaussian Splatting as Markov Chain Monte Carlo [30.04096439325343]
  3D Gaussian Splatting has recently become popular for neural rendering. We rethink the set of 3D Gaussians as a random sample drawn from an underlying probability distribution. We introduce a regularizer that promotes the removal of unused Gaussians.
  arXiv  Detail & Related papers  (2024-04-15T09:01:47Z)
• Neural Boltzmann Machines [2.179313476241343]
  Conditional generative models are capable of using contextual information as input to create new imaginative outputs. Conditional Restricted Boltzmann Machines (CRBMs) are one class of conditional generative models that have proven to be especially adept at modeling noisy discrete or continuous data. We generalize CRBMs by converting each of the CRBM parameters to their own neural networks that are allowed to be functions of the conditional inputs.
  arXiv  Detail & Related papers  (2023-05-15T04:03:51Z)
• Joint Generator-Ranker Learning for Natural Language Generation [99.16268050116717]
  JGR is a novel joint training algorithm that integrates the generator and the ranker in a single framework. By iteratively updating the generator and the ranker, JGR can effectively harmonize their learning and enhance their quality jointly.
  arXiv  Detail & Related papers  (2022-06-28T12:58:30Z)
• Langevin Monte Carlo for Contextual Bandits [72.00524614312002]
  Langevin Monte Carlo Thompson Sampling (LMC-TS) is proposed to directly sample from the posterior distribution in contextual bandits. We prove that the proposed algorithm achieves the same sublinear regret bound as the best Thompson sampling algorithms for a special case of contextual bandits.
  arXiv  Detail & Related papers  (2022-06-22T17:58:23Z)
• Distributionally Robust Models with Parametric Likelihood Ratios [123.05074253513935]
  Three simple ideas allow us to train models with DRO using a broader class of parametric likelihood ratios. We find that models trained with the resulting parametric adversaries are consistently more robust to subpopulation shifts when compared to other DRO approaches.
  arXiv  Detail & Related papers  (2022-04-13T12:43:12Z)
• Barriers and Dynamical Paths in Alternating Gibbs Sampling of Restricted Boltzmann Machines [0.0]
  We study the performance of Alternating Gibbs Sampling (AGS) on several analytically tractable models. We show that standard AGS is not more efficient than classical Metropolis-Hastings (MH) sampling of the effective energy landscape. We illustrate our findings on three datasets: Bars and Stripes and MNIST, well known in machine learning, and the so-called Lattice Proteins.
  arXiv  Detail & Related papers  (2021-07-13T12:07:56Z)
• ReMix: Towards Image-to-Image Translation with Limited Data [154.71724970593036]
  We propose a data augmentation method (ReMix) to tackle this issue. We interpolate training samples at the feature level and propose a novel content loss based on the perceptual relations among samples. The proposed approach effectively reduces the ambiguity of generation and renders content-preserving results.
  arXiv  Detail & Related papers  (2021-03-31T06:24:10Z)
• Learning Gaussian-Bernoulli RBMs using Difference of Convex Functions Optimization [0.9137554315375919]
  We show that negative log-likelihood for a GB-RBM can be expressed as a difference of convex functions. We propose a difference of convex functions programming (S-DCP) algorithm for learning the GB-RBM. It is seen that S-DCP is better than the CD and PCD algorithms in terms of speed of learning and the quality of the generative model learnt.
  arXiv  Detail & Related papers  (2021-02-11T19:15:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.