BNEM: A Boltzmann Sampler Based on Bootstrapped Noised Energy Matching

• URL: http://arxiv.org/abs/2409.09787v2
• Date: Thu, 10 Oct 2024 14:09:44 GMT
• Title: BNEM: A Boltzmann Sampler Based on Bootstrapped Noised Energy Matching
• Authors: RuiKang OuYang, Bo Qiang, José Miguel Hernández-Lobato,
• Abstract summary: We learn neural samplers given energy functions instead of data sampled from the Boltzmann distribution. We propose a diffusion-based sampler, Noised Energy Matching, which theoretically has lower variance and more complexity. The experimental results demonstrate that BNEM can achieve state-of-the-art performance while being more robust.
• Score: 29.531531253864753
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Developing an efficient sampler capable of generating independent and identically distributed (IID) samples from a Boltzmann distribution is a crucial challenge in scientific research, e.g. molecular dynamics. In this work, we intend to learn neural samplers given energy functions instead of data sampled from the Boltzmann distribution. By learning the energies of the noised data, we propose a diffusion-based sampler, Noised Energy Matching, which theoretically has lower variance and more complexity compared to related works. Furthermore, a novel bootstrapping technique is applied to NEM to balance between bias and variance. We evaluate NEM and BNEM on a 2-dimensional 40 Gaussian Mixture Model (GMM) and a 4-particle double-well potential (DW-4). The experimental results demonstrate that BNEM can achieve state-of-the-art performance while being more robust.

Related papers

• Efficient and Unbiased Sampling of Boltzmann Distributions via Consistency Models [28.596110110641327]
  This paper introduces a novel sampling method that effectively combines Consistency Models (CMs) with importance sampling. We evaluate our approach on both synthetic energy functions and equivariant n-body particle systems. Our method produces unbiased samples using only 6-25 NFEs while achieving a comparable Effective Sample Size (ESS) to Denoising Diffusion Probabilistic Models (DDPMs) that require approximately 100 NFEs.
  arXiv  Detail & Related papers  (2024-09-11T15:02:52Z)
• Discrete generative diffusion models without stochastic differential equations: a tensor network approach [1.5839621757142595]
  Diffusion models (DMs) are a class of generative machine learning methods. We show how to use networks (TNs) to efficiently define and sample such discrete models''
  arXiv  Detail & Related papers  (2024-07-15T18:00:11Z)
• Theoretical Insights for Diffusion Guidance: A Case Study for Gaussian Mixture Models [59.331993845831946]
  Diffusion models benefit from instillation of task-specific information into the score function to steer the sample generation towards desired properties. This paper provides the first theoretical study towards understanding the influence of guidance on diffusion models in the context of Gaussian mixture models.
  arXiv  Detail & Related papers  (2024-03-03T23:15:48Z)
• Iterated Denoising Energy Matching for Sampling from Boltzmann Densities [109.23137009609519]
  Iterated Denoising Energy Matching (iDEM) iDEM alternates between (I) sampling regions of high model density from a diffusion-based sampler and (II) using these samples in our matching objective. We show that the proposed approach achieves state-of-the-art performance on all metrics and trains $2-5times$ faster.
  arXiv  Detail & Related papers  (2024-02-09T01:11:23Z)
• Generalized Contrastive Divergence: Joint Training of Energy-Based Model and Diffusion Model through Inverse Reinforcement Learning [13.22531381403974]
  Generalized Contrastive Divergence (GCD) is a novel objective function for training an energy-based model (EBM) and a sampler simultaneously. We present preliminary yet promising results showing that joint training is beneficial for both EBM and a diffusion model.
  arXiv  Detail & Related papers  (2023-12-06T10:10:21Z)
• Gaussian Mixture Solvers for Diffusion Models [84.83349474361204]
  We introduce a novel class of SDE-based solvers called GMS for diffusion models. Our solver outperforms numerous SDE-based solvers in terms of sample quality in image generation and stroke-based synthesis.
  arXiv  Detail & Related papers  (2023-11-02T02:05:38Z)
• IBADR: an Iterative Bias-Aware Dataset Refinement Framework for Debiasing NLU models [52.03761198830643]
  We propose IBADR, an Iterative Bias-Aware dataset Refinement framework. We first train a shallow model to quantify the bias degree of samples in the pool. Then, we pair each sample with a bias indicator representing its bias degree, and use these extended samples to train a sample generator. In this way, this generator can effectively learn the correspondence relationship between bias indicators and samples.
  arXiv  Detail & Related papers  (2023-11-01T04:50:38Z)
• Discrete Diffusion Modeling by Estimating the Ratios of the Data Distribution [67.9215891673174]
  We propose score entropy as a novel loss that naturally extends score matching to discrete spaces. We test our Score Entropy Discrete Diffusion models on standard language modeling tasks.
  arXiv  Detail & Related papers  (2023-10-25T17:59:12Z)
• Dual Training of Energy-Based Models with Overparametrized Shallow Neural Networks [41.702175127106784]
  Energy-based models (EBMs) are generative models that are usually trained via maximum likelihood estimation. We propose a dual formulation of an EBMs algorithm in which the particles are sometimes restarted at random samples drawn from the data set, and show that performing these restarts corresponds to a score every step. These results are illustrated in simple numerical experiments.
  arXiv  Detail & Related papers  (2021-07-11T21:43:18Z)
• Energy-Based Reranking: Improving Neural Machine Translation Using Energy-Based Models [59.039592890187144]
  We study the discrepancy between maximum likelihood estimation (MLE) and task measures such as BLEU score for autoregressive neural machine translation (NMT) Samples drawn from an MLE-based trained NMT support the desired distribution -- there are samples with much higher BLEU score compared to the beam decoding output. We use both marginal energy models (over target sentence) and joint energy models (over both source and target sentences) to improve our algorithm.
  arXiv  Detail & Related papers  (2020-09-20T02:50:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.