PFGM++: Unlocking the Potential of Physics-Inspired Generative Models

• URL: http://arxiv.org/abs/2302.04265v2
• Date: Fri, 10 Feb 2023 16:45:02 GMT
• Title: PFGM++: Unlocking the Potential of Physics-Inspired Generative Models
• Authors: Yilun Xu, Ziming Liu, Yonglong Tian, Shangyuan Tong, Max Tegmark, Tommi Jaakkola
• Abstract summary: We introduce a new family of physics-inspired generative models termed PFGM++. These models realize generative trajectories for $N$ dimensional data by embedding paths in $N+D$ dimensional space. We show that models with finite $D$ can be superior to previous state-of-the-art diffusion models.
• Score: 14.708385906024546
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We introduce a new family of physics-inspired generative models termed PFGM++ that unifies diffusion models and Poisson Flow Generative Models (PFGM). These models realize generative trajectories for $N$ dimensional data by embedding paths in $N{+}D$ dimensional space while still controlling the progression with a simple scalar norm of the $D$ additional variables. The new models reduce to PFGM when $D{=}1$ and to diffusion models when $D{\to}\infty$. The flexibility of choosing $D$ allows us to trade off robustness against rigidity as increasing $D$ results in more concentrated coupling between the data and the additional variable norms. We dispense with the biased large batch field targets used in PFGM and instead provide an unbiased perturbation-based objective similar to diffusion models. To explore different choices of $D$, we provide a direct alignment method for transferring well-tuned hyperparameters from diffusion models ($D{\to} \infty$) to any finite $D$ values. Our experiments show that models with finite $D$ can be superior to previous state-of-the-art diffusion models on CIFAR-10/FFHQ $64{\times}64$ datasets, with FID scores of $1.91/2.43$ when $D{=}2048/128$. In class-conditional setting, $D{=}2048$ yields current state-of-the-art FID of $1.74$ on CIFAR-10. In addition, we demonstrate that models with smaller $D$ exhibit improved robustness against modeling errors. Code is available at https://github.com/Newbeeer/pfgmpp

Related papers

• Scaling Laws in Linear Regression: Compute, Parameters, and Data [86.48154162485712]
  We study the theory of scaling laws in an infinite dimensional linear regression setup. We show that the reducible part of the test error is $Theta(-(a-1) + N-(a-1)/a)$. Our theory is consistent with the empirical neural scaling laws and verified by numerical simulation.
  arXiv  Detail & Related papers  (2024-06-12T17:53:29Z)
• Towards Faster Non-Asymptotic Convergence for Diffusion-Based Generative Models [49.81937966106691]
  We develop a suite of non-asymptotic theory towards understanding the data generation process of diffusion models. In contrast to prior works, our theory is developed based on an elementary yet versatile non-asymptotic approach.
  arXiv  Detail & Related papers  (2023-06-15T16:30:08Z)
• Better Diffusion Models Further Improve Adversarial Training [97.44991845907708]
  It has been recognized that the data generated by the diffusion probabilistic model (DDPM) improves adversarial training. This paper gives an affirmative answer by employing the most recent diffusion model which has higher efficiency. Our adversarially trained models achieve state-of-the-art performance on RobustBench using only generated data.
  arXiv  Detail & Related papers  (2023-02-09T13:46:42Z)
• Poisson Flow Generative Models [9.843778728210427]
  "Poisson flow" generative model maps a uniform distribution on a high-dimensional hemisphere into any data distribution. PFGM achieves current state-of-the-art performance among the normalizing flow models on CIFAR-10.
  arXiv  Detail & Related papers  (2022-09-22T17:26:58Z)
• Neural Implicit Manifold Learning for Topology-Aware Density Estimation [15.878635603835063]
  Current generative models learn $mathcalM$ by mapping an $m$-dimensional latent variable through a neural network. We show that our model can learn manifold-supported distributions with complex topologies more accurately than pushforward models.
  arXiv  Detail & Related papers  (2022-06-22T18:00:00Z)
• Full-Span Log-Linear Model and Fast Learning Algorithm [1.9798034349981162]
  The full-span log-linear(FSLL) model is considered an $n$-th order Boltzmann machine. The FSLL model has $|X|-1$ parameters and can represent arbitrary positive distributions of $X$. Experiments presented that the FSLL successfully learned six training datasets such that $|X|=220$ within one minute with a laptop PC.
  arXiv  Detail & Related papers  (2022-02-17T06:51:59Z)
• Lessons from $O(N)$ models in one dimension [0.0]
  Various topics related to the $O(N)$ model in one spacetime dimension (i.e. ordinary quantum mechanics) are considered. The focus is on a pedagogical presentation of quantum field theory methods in a simpler context.
  arXiv  Detail & Related papers  (2021-09-14T11:36:30Z)
• Exploring Sparse Expert Models and Beyond [51.90860155810848]
  Mixture-of-Experts (MoE) models can achieve promising results with outrageous large amount of parameters but constant computation cost. We propose a simple method called expert prototyping that splits experts into different prototypes and applies $k$ top-$1$ routing. This strategy improves the model quality but maintains constant computational costs, and our further exploration on extremely large-scale models reflects that it is more effective in training larger models.
  arXiv  Detail & Related papers  (2021-05-31T16:12:44Z)
• Adversarial robustness against multiple $l_p$-threat models at the price of one and how to quickly fine-tune robust models to another threat model [79.05253587566197]
  Adrial training (AT) in order to achieve adversarial robustness wrt single $l_p$-threat models has been discussed extensively. In this paper we develop a simple and efficient training scheme to achieve adversarial robustness against the union of $l_p$-threat models.
  arXiv  Detail & Related papers  (2021-05-26T12:20:47Z)
• Improving Robustness and Generality of NLP Models Using Disentangled Representations [62.08794500431367]
  Supervised neural networks first map an input $x$ to a single representation $z$, and then map $z$ to the output label $y$. We present methods to improve robustness and generality of NLP models from the standpoint of disentangled representation learning. We show that models trained with the proposed criteria provide better robustness and domain adaptation ability in a wide range of supervised learning tasks.
  arXiv  Detail & Related papers  (2020-09-21T02:48:46Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.