Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent

• URL: http://arxiv.org/abs/2106.07832v1
• Date: Tue, 15 Jun 2021 01:35:17 GMT
• Title: Learning Equivariant Energy Based Models with Equivariant Stein Variational Gradient Descent
• Authors: Priyank Jaini, Lars Holdijk and Max Welling
• Abstract summary: We focus on the problem of efficient sampling and learning of probability densities by incorporating symmetries in probabilistic models. We first introduce Equivariant Stein Variational Gradient Descent algorithm -- an equivariant sampling method based on Stein's identity for sampling from densities with symmetries. We propose new ways of improving and scaling up training of energy based models.
• Score: 80.73580820014242
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We focus on the problem of efficient sampling and learning of probability densities by incorporating symmetries in probabilistic models. We first introduce Equivariant Stein Variational Gradient Descent algorithm -- an equivariant sampling method based on Stein's identity for sampling from densities with symmetries. Equivariant SVGD explicitly incorporates symmetry information in a density through equivariant kernels which makes the resultant sampler efficient both in terms of sample complexity and the quality of generated samples. Subsequently, we define equivariant energy based models to model invariant densities that are learned using contrastive divergence. By utilizing our equivariant SVGD for training equivariant EBMs, we propose new ways of improving and scaling up training of energy based models. We apply these equivariant energy models for modelling joint densities in regression and classification tasks for image datasets, many-body particle systems and molecular structure generation.

Related papers

• Equivariant score-based generative models provably learn distributions with symmetries efficiently [7.90752151686317]
  Empirical studies have demonstrated that incorporating symmetries into generative models can provide better generalization and sampling efficiency. We provide the first theoretical analysis and guarantees of score-based generative models (SGMs) for learning distributions that are invariant with respect to some group symmetry.
  arXiv  Detail & Related papers  (2024-10-02T05:14:28Z)
• Weak Generative Sampler to Efficiently Sample Invariant Distribution of Stochastic Differential Equation [8.67581853745823]
  Current deep learning-based method solves the stationary Fokker--Planck equation to determine the invariant probability density function in form of deep neural networks. We introduce a framework that employs a weak generative sampler (WGS) to directly generate independent and identically distributed (iid) samples. Our proposed loss function is based on the weak form of the Fokker--Planck equation, integrating normalizing flows to characterize the invariant distribution.
  arXiv  Detail & Related papers  (2024-05-29T16:41:42Z)
• On the Trajectory Regularity of ODE-based Diffusion Sampling [79.17334230868693]
  Diffusion-based generative models use differential equations to establish a smooth connection between a complex data distribution and a tractable prior distribution. In this paper, we identify several intriguing trajectory properties in the ODE-based sampling process of diffusion models.
  arXiv  Detail & Related papers  (2024-05-18T15:59:41Z)
• Geometric Neural Diffusion Processes [55.891428654434634]
  We extend the framework of diffusion models to incorporate a series of geometric priors in infinite-dimension modelling. We show that with these conditions, the generative functional model admits the same symmetry.
  arXiv  Detail & Related papers  (2023-07-11T16:51:38Z)
• Learning Symmetric Embeddings for Equivariant World Models [9.781637768189158]
  We propose learning symmetric embedding networks (SENs) that encode an input space (e.g. images) This network can be trained end-to-end with an equivariant task network to learn an explicitly symmetric representation. Our experiments demonstrate that SENs facilitate the application of equivariant networks to data with complex symmetry representations.
  arXiv  Detail & Related papers  (2022-04-24T22:31:52Z)
• Equivariance Discovery by Learned Parameter-Sharing [153.41877129746223]
  We study how to discover interpretable equivariances from data. Specifically, we formulate this discovery process as an optimization problem over a model's parameter-sharing schemes. Also, we theoretically analyze the method for Gaussian data and provide a bound on the mean squared gap between the studied discovery scheme and the oracle scheme.
  arXiv  Detail & Related papers  (2022-04-07T17:59:19Z)
• Equivariant Diffusion for Molecule Generation in 3D [74.289191525633]
  This work introduces a diffusion model for molecule computation generation in 3D that is equivariant to Euclidean transformations. Experimentally, the proposed method significantly outperforms previous 3D molecular generative methods regarding the quality of generated samples and efficiency at training time.
  arXiv  Detail & Related papers  (2022-03-31T12:52:25Z)
• Equivariant vector field network for many-body system modeling [65.22203086172019]
  Equivariant Vector Field Network (EVFN) is built on a novel equivariant basis and the associated scalarization and vectorization layers. We evaluate our method on predicting trajectories of simulated Newton mechanics systems with both full and partially observed data.
  arXiv  Detail & Related papers  (2021-10-26T14:26:25Z)
• Equivariant Learning of Stochastic Fields: Gaussian Processes and Steerable Conditional Neural Processes [44.51932024971217]
  We study the problem of learning fields, i.e. processes whose samples are fields like those occurring in physics and engineering. We introduce Steerable Conditional Neural Processes (SteerCNPs), a new, fully equivariant member of the Neural Process family. In experiments with Gaussian process vector fields, images, and real-world weather data, we observe that SteerCNPs significantly improve the performance of previous models.
  arXiv  Detail & Related papers  (2020-11-25T18:00:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.