Trivialized Momentum Facilitates Diffusion Generative Modeling on Lie Groups

• URL: http://arxiv.org/abs/2405.16381v2
• Date: Wed, 12 Feb 2025 00:48:57 GMT
• Title: Trivialized Momentum Facilitates Diffusion Generative Modeling on Lie Groups
• Authors: Yuchen Zhu, Tianrong Chen, Lingkai Kong, Evangelos A. Theodorou, Molei Tao,
• Abstract summary: This article demonstrates how a technique called trivialization' can transfer the effectiveness of diffusion models in Euclidean spaces to Lie groups. It achieves state-of-the-art performance on protein and RNA torsion angle generation and sophisticated torus datasets.
• Score: 37.78638937228254
• License:
• Abstract: The generative modeling of data on manifolds is an important task, for which diffusion models in flat spaces typically need nontrivial adaptations. This article demonstrates how a technique called `trivialization' can transfer the effectiveness of diffusion models in Euclidean spaces to Lie groups. In particular, an auxiliary momentum variable was algorithmically introduced to help transport the position variable between data distribution and a fixed, easy-to-sample distribution. Normally, this would incur further difficulty for manifold data because momentum lives in a space that changes with the position. However, our trivialization technique creates a new momentum variable that stays in a simple fixed vector space. This design, together with a manifold preserving integrator, simplifies implementation and avoids inaccuracies created by approximations such as projections to tangent space and manifold, which were typically used in prior work, hence facilitating generation with high-fidelity and efficiency. The resulting method achieves state-of-the-art performance on protein and RNA torsion angle generation and sophisticated torus datasets. We also, arguably for the first time, tackle the generation of data on high-dimensional Special Orthogonal and Unitary groups, the latter essential for quantum problems. Code is available at https://github.com/yuchen-zhu-zyc/TDM.

Related papers

• Robust training of implicit generative models for multivariate and heavy-tailed distributions with an invariant statistical loss [0.4249842620609682]
  We build on the textitinvariant statistical loss (ISL) method introduced in citede2024training. We extend it to handle heavy-tailed and multivariate data distributions. We assess its performance in generative generative modeling and explore its potential as a pretraining technique for generative adversarial networks (GANs)
  arXiv  Detail & Related papers  (2024-10-29T10:27:50Z)
• Distributional Reduction: Unifying Dimensionality Reduction and Clustering with Gromov-Wasserstein [56.62376364594194]
  Unsupervised learning aims to capture the underlying structure of potentially large and high-dimensional datasets. In this work, we revisit these approaches under the lens of optimal transport and exhibit relationships with the Gromov-Wasserstein problem. This unveils a new general framework, called distributional reduction, that recovers DR and clustering as special cases and allows addressing them jointly within a single optimization problem.
  arXiv  Detail & Related papers  (2024-02-03T19:00:19Z)
• Scaling Riemannian Diffusion Models [68.52820280448991]
  We show that our method enables us to scale to high dimensional tasks on nontrivial manifold. We model QCD densities on $SU(n)$ lattices and contrastively learned embeddings on high dimensional hyperspheres.
  arXiv  Detail & Related papers  (2023-10-30T21:27:53Z)
• Geometrically Aligned Transfer Encoder for Inductive Transfer in Regression Tasks [5.038936775643437]
  We propose a novel transfer technique based on differential geometry, namely the Geometrically Aligned Transfer (GATE) We find a proper diffeomorphism between pairs of tasks to ensure that every arbitrary point maps to a locally flat coordinate in the overlapping region, allowing the transfer of knowledge from the source to the target data. GATE outperforms conventional methods and exhibits stable behavior in both the latent space and extrapolation regions for various molecular graph datasets.
  arXiv  Detail & Related papers  (2023-10-10T07:11:25Z)
• Disentanglement via Latent Quantization [60.37109712033694]
  In this work, we construct an inductive bias towards encoding to and decoding from an organized latent space. We demonstrate the broad applicability of this approach by adding it to both basic data-re (vanilla autoencoder) and latent-reconstructing (InfoGAN) generative models.
  arXiv  Detail & Related papers  (2023-05-28T06:30:29Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• The Manifold Scattering Transform for High-Dimensional Point Cloud Data [16.500568323161563]
  We present practical schemes for implementing the manifold scattering transform to datasets arising in naturalistic systems. We show that our methods are effective for signal classification and manifold classification tasks.
  arXiv  Detail & Related papers  (2022-06-21T02:15:00Z)
• Generation of data on discontinuous manifolds via continuous stochastic non-invertible networks [6.201770337181472]
  We show how to generate discontinuous distributions using continuous networks. We derive a link between the cost functions and the information-theoretic formulation. We apply our approach to synthetic 2D distributions to demonstrate both reconstruction and generation of discontinuous distributions.
  arXiv  Detail & Related papers  (2021-12-17T17:39:59Z)
• Learning High-Dimensional Distributions with Latent Neural Fokker-Planck Kernels [67.81799703916563]
  We introduce new techniques to formulate the problem as solving Fokker-Planck equation in a lower-dimensional latent space. Our proposed model consists of latent-distribution morphing, a generator and a parameterized Fokker-Planck kernel function.
  arXiv  Detail & Related papers  (2021-05-10T17:42:01Z)
• Generative Model without Prior Distribution Matching [26.91643368299913]
  Variational Autoencoder (VAE) and its variations are classic generative models by learning a low-dimensional latent representation to satisfy some prior distribution. We propose to let the prior match the embedding distribution rather than imposing the latent variables to fit the prior.
  arXiv  Detail & Related papers  (2020-09-23T09:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.