Sampling from Discrete Energy-Based Models with Quality/Efficiency Trade-offs

• URL: http://arxiv.org/abs/2112.05702v1
• Date: Fri, 10 Dec 2021 17:51:37 GMT
• Title: Sampling from Discrete Energy-Based Models with Quality/Efficiency Trade-offs
• Authors: Bryan Eikema, Germ\'an Kruszewski, Hady Elsahar, Marc Dymetman
• Abstract summary: Energy-Based Models (EBMs) allow for extremely flexible specifications of probability distributions. They do not provide a mechanism for obtaining exact samples from these distributions. We propose a new approximate sampling technique, Quasi Rejection Sampling (QRS), that allows for a trade-off between sampling efficiency and sampling quality.
• Score: 3.491202838583993
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Energy-Based Models (EBMs) allow for extremely flexible specifications of probability distributions. However, they do not provide a mechanism for obtaining exact samples from these distributions. Monte Carlo techniques can aid us in obtaining samples if some proposal distribution that we can easily sample from is available. For instance, rejection sampling can provide exact samples but is often difficult or impossible to apply due to the need to find a proposal distribution that upper-bounds the target distribution everywhere. Approximate Markov chain Monte Carlo sampling techniques like Metropolis-Hastings are usually easier to design, exploiting a local proposal distribution that performs local edits on an evolving sample. However, these techniques can be inefficient due to the local nature of the proposal distribution and do not provide an estimate of the quality of their samples. In this work, we propose a new approximate sampling technique, Quasi Rejection Sampling (QRS), that allows for a trade-off between sampling efficiency and sampling quality, while providing explicit convergence bounds and diagnostics. QRS capitalizes on the availability of high-quality global proposal distributions obtained from deep learning models. We demonstrate the effectiveness of QRS sampling for discrete EBMs over text for the tasks of controlled text generation with distributional constraints and paraphrase generation. We show that we can sample from such EBMs with arbitrary precision at the cost of sampling efficiency.

Related papers

• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional general score-mismatched diffusion samplers. We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions. This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• Conditional sampling within generative diffusion models [12.608803080528142]
  We present a review of existing computational approaches to conditional sampling within generative diffusion models. We highlight key methodologies that either utilise the joint distribution, or rely on (pre-trained) marginal distributions with explicit likelihoods.
  arXiv  Detail & Related papers  (2024-09-15T07:48:40Z)
• Iterated Denoising Energy Matching for Sampling from Boltzmann Densities [109.23137009609519]
  Iterated Denoising Energy Matching (iDEM) iDEM alternates between (I) sampling regions of high model density from a diffusion-based sampler and (II) using these samples in our matching objective. We show that the proposed approach achieves state-of-the-art performance on all metrics and trains $2-5times$ faster.
  arXiv  Detail & Related papers  (2024-02-09T01:11:23Z)
• A Block Metropolis-Hastings Sampler for Controllable Energy-based Text Generation [78.81021361497311]
  We develop a novel Metropolis-Hastings (MH) sampler that proposes re-writes of the entire sequence in each step via iterative prompting of a large language model. Our new sampler allows for more efficient and accurate sampling from a target distribution and (b) allows generation length to be determined through the sampling procedure rather than fixed in advance.
  arXiv  Detail & Related papers  (2023-12-07T18:30:15Z)
• Conditional Sampling of Variational Autoencoders via Iterated Approximate Ancestral Sampling [7.357511266926065]
  Conditional sampling of variational autoencoders (VAEs) is needed in various applications, such as missing data imputation, but is computationally intractable. A principled choice forally exact conditional sampling is Metropolis-within-Gibbs (MWG)
  arXiv  Detail & Related papers  (2023-08-17T16:08:18Z)
• LSB: Local Self-Balancing MCMC in Discrete Spaces [2.385916960125935]
  This work considers using machine learning to adapt the proposal distribution to the target, in order to improve the sampling efficiency in the purely discrete domain. We call the resulting sampler as the Locally Self-Balancing Sampler (LSB)
  arXiv  Detail & Related papers  (2021-09-08T18:31:26Z)
• Reparameterized Sampling for Generative Adversarial Networks [71.30132908130581]
  We propose REP-GAN, a novel sampling method that allows general dependent proposals by REizing the Markov chains into the latent space of the generator. Empirically, extensive experiments on synthetic and real datasets demonstrate that our REP-GAN largely improves the sample efficiency and obtains better sample quality simultaneously.
  arXiv  Detail & Related papers  (2021-07-01T10:34:55Z)
• Uncorrelated problem-specific samples of quantum states from zero-mean Wishart distributions [4.289102530380288]
  We present a two-step algorithm for sampling from the quantum state space. We establish the explicit form of the induced Wishart distribution for quantum states. We demonstrate that this sampling algorithm is very efficient for one-qubit and two-qubit states, and reasonably efficient for three-qubit states.
  arXiv  Detail & Related papers  (2021-06-16T03:06:41Z)
• Exposing the Implicit Energy Networks behind Masked Language Models via Metropolis--Hastings [57.133639209759615]
  We interpret sequences as energy-based sequence models and propose two energy parametrizations derivable from traineds. We develop a tractable emph scheme based on the Metropolis-Hastings Monte Carlo algorithm. We validate the effectiveness of the proposed parametrizations by exploring the quality of samples drawn from these energy-based models.
  arXiv  Detail & Related papers  (2021-06-04T22:04:30Z)
• Learning Energy-Based Models by Diffusion Recovery Likelihood [61.069760183331745]
  We present a diffusion recovery likelihood method to tractably learn and sample from a sequence of energy-based models. After training, synthesized images can be generated by the sampling process that initializes from Gaussian white noise distribution. On unconditional CIFAR-10 our method achieves FID 9.58 and inception score 8.30, superior to the majority of GANs.
  arXiv  Detail & Related papers  (2020-12-15T07:09:02Z)
• A Neural Network MCMC sampler that maximizes Proposal Entropy [3.4698840925433765]
  Augmenting samplers with neural networks can potentially improve their efficiency. Our network architecture utilizes the gradient of the target distribution for generating proposals. The adaptive sampler achieves unbiased sampling with significantly higher proposal entropy than Langevin dynamics sampler.
  arXiv  Detail & Related papers  (2020-10-07T18:01:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.