Trivialized Momentum Facilitates Diffusion Generative Modeling on Lie Groups

• URL: http://arxiv.org/abs/2405.16381v1
• Date: Sat, 25 May 2024 23:53:07 GMT
• Title: Trivialized Momentum Facilitates Diffusion Generative Modeling on Lie Groups
• Authors: Yuchen Zhu, Tianrong Chen, Lingkai Kong, Evangelos A. Theodorou, Molei Tao,
• Abstract summary: This article demonstrates how a technique called trivialization' can transfer the effectiveness of diffusion models in Euclidean spaces to Lie groups. A momentum variable was algorithmically introduced to help transport the position variable between data distribution and a fixed, easy-to-sample distribution. The resulting method achieves state-of-the-art performance on protein and RNA torsion angle generation and sophisticated torus datasets.
• Score: 37.78638937228254
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The generative modeling of data on manifold is an important task, for which diffusion models in flat spaces typically need nontrivial adaptations. This article demonstrates how a technique called `trivialization' can transfer the effectiveness of diffusion models in Euclidean spaces to Lie groups. In particular, an auxiliary momentum variable was algorithmically introduced to help transport the position variable between data distribution and a fixed, easy-to-sample distribution. Normally, this would incur further difficulty for manifold data because momentum lives in a space that changes with the position. However, our trivialization technique creates to a new momentum variable that stays in a simple $\textbf{fixed vector space}$. This design, together with a manifold preserving integrator, simplifies implementation and avoids inaccuracies created by approximations such as projections to tangent space and manifold, which were typically used in prior work, hence facilitating generation with high-fidelity and efficiency. The resulting method achieves state-of-the-art performance on protein and RNA torsion angle generation and sophisticated torus datasets. We also, arguably for the first time, tackle the generation of data on high-dimensional Special Orthogonal and Unitary groups, the latter essential for quantum problems.

Related papers

• Linear combinations of Gaussian latents in generative models: interpolation and beyond [6.38754204972456]
  Combination of Gaussian variables (COG) is a general purpose method that is easy to implement yet outperforms recent sophisticated methods. COG naturally addresses the broader task of forming general linear combinations of latent variables, allowing the construction of subspaces of the latent space.
  arXiv  Detail & Related papers  (2024-08-16T06:43:58Z)
• Learning Divergence Fields for Shift-Robust Graph Representations [73.11818515795761]
  In this work, we propose a geometric diffusion model with learnable divergence fields for the challenging problem with interdependent data. We derive a new learning objective through causal inference, which can guide the model to learn generalizable patterns of interdependence that are insensitive across domains.
  arXiv  Detail & Related papers  (2024-06-07T14:29:21Z)
• Distributional Reduction: Unifying Dimensionality Reduction and Clustering with Gromov-Wasserstein [56.62376364594194]
  Unsupervised learning aims to capture the underlying structure of potentially large and high-dimensional datasets. In this work, we revisit these approaches under the lens of optimal transport and exhibit relationships with the Gromov-Wasserstein problem. This unveils a new general framework, called distributional reduction, that recovers DR and clustering as special cases and allows addressing them jointly within a single optimization problem.
  arXiv  Detail & Related papers  (2024-02-03T19:00:19Z)
• Geometrically Aligned Transfer Encoder for Inductive Transfer in Regression Tasks [5.038936775643437]
  We propose a novel transfer technique based on differential geometry, namely the Geometrically Aligned Transfer (GATE) We find a proper diffeomorphism between pairs of tasks to ensure that every arbitrary point maps to a locally flat coordinate in the overlapping region, allowing the transfer of knowledge from the source to the target data. GATE outperforms conventional methods and exhibits stable behavior in both the latent space and extrapolation regions for various molecular graph datasets.
  arXiv  Detail & Related papers  (2023-10-10T07:11:25Z)
• Random Smoothing Regularization in Kernel Gradient Descent Learning [24.383121157277007]
  We present a framework for random smoothing regularization that can adaptively learn a wide range of ground truth functions belonging to the classical Sobolev spaces. Our estimator can adapt to the structural assumptions of the underlying data and avoid the curse of dimensionality.
  arXiv  Detail & Related papers  (2023-05-05T13:37:34Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• DIFFormer: Scalable (Graph) Transformers Induced by Energy Constrained Diffusion [66.21290235237808]
  We introduce an energy constrained diffusion model which encodes a batch of instances from a dataset into evolutionary states. We provide rigorous theory that implies closed-form optimal estimates for the pairwise diffusion strength among arbitrary instance pairs. Experiments highlight the wide applicability of our model as a general-purpose encoder backbone with superior performance in various tasks.
  arXiv  Detail & Related papers  (2023-01-23T15:18:54Z)
• The Manifold Scattering Transform for High-Dimensional Point Cloud Data [16.500568323161563]
  We present practical schemes for implementing the manifold scattering transform to datasets arising in naturalistic systems. We show that our methods are effective for signal classification and manifold classification tasks.
  arXiv  Detail & Related papers  (2022-06-21T02:15:00Z)
• Learning High-Dimensional Distributions with Latent Neural Fokker-Planck Kernels [67.81799703916563]
  We introduce new techniques to formulate the problem as solving Fokker-Planck equation in a lower-dimensional latent space. Our proposed model consists of latent-distribution morphing, a generator and a parameterized Fokker-Planck kernel function.
  arXiv  Detail & Related papers  (2021-05-10T17:42:01Z)
• Generative Model without Prior Distribution Matching [26.91643368299913]
  Variational Autoencoder (VAE) and its variations are classic generative models by learning a low-dimensional latent representation to satisfy some prior distribution. We propose to let the prior match the embedding distribution rather than imposing the latent variables to fit the prior.
  arXiv  Detail & Related papers  (2020-09-23T09:33:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.