Understanding the Local Geometry of Generative Model Manifolds

• URL: http://arxiv.org/abs/2408.08307v1
• Date: Thu, 15 Aug 2024 17:59:06 GMT
• Title: Understanding the Local Geometry of Generative Model Manifolds
• Authors: Ahmed Imtiaz Humayun, Ibtihel Amara, Candice Schumann, Golnoosh Farnadi, Negar Rostamzadeh, Mohammad Havaei,
• Abstract summary: We study the relationship between the textitlocal geometry of the learned manifold and downstream generation. We provide quantitative and qualitative evidence showing that for a given latent, the local descriptors are correlated with generation aesthetics, artifacts, uncertainty, and even memorization.
• Score: 14.191548577311904
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Deep generative models learn continuous representations of complex data manifolds using a finite number of samples during training. For a pre-trained generative model, the common way to evaluate the quality of the manifold representation learned, is by computing global metrics like Fr\'echet Inception Distance using a large number of generated and real samples. However, generative model performance is not uniform across the learned manifold, e.g., for \textit{foundation models} like Stable Diffusion generation performance can vary significantly based on the conditioning or initial noise vector being denoised. In this paper we study the relationship between the \textit{local geometry of the learned manifold} and downstream generation. Based on the theory of continuous piecewise-linear (CPWL) generators, we use three geometric descriptors - scaling ($\psi$), rank ($\nu$), and complexity ($\delta$) - to characterize a pre-trained generative model manifold locally. We provide quantitative and qualitative evidence showing that for a given latent, the local descriptors are correlated with generation aesthetics, artifacts, uncertainty, and even memorization. Finally we demonstrate that training a \textit{reward model} on the local geometry can allow controlling the likelihood of a generated sample under the learned distribution.

Related papers

• SpaceMesh: A Continuous Representation for Learning Manifold Surface Meshes [61.110517195874074]
  We present a scheme to directly generate manifold, polygonal meshes of complex connectivity as the output of a neural network. Our key innovation is to define a continuous latent connectivity space at each mesh, which implies the discrete mesh. In applications, this approach not only yields high-quality outputs from generative models, but also enables directly learning challenging geometry processing tasks such as mesh repair.
  arXiv  Detail & Related papers  (2024-09-30T17:59:03Z)
• Heat Death of Generative Models in Closed-Loop Learning [63.83608300361159]
  We study the learning dynamics of generative models that are fed back their own produced content in addition to their original training dataset. We show that, unless a sufficient amount of external data is introduced at each iteration, any non-trivial temperature leads the model to degenerate.
  arXiv  Detail & Related papers  (2024-04-02T21:51:39Z)
• Improving Out-of-Distribution Robustness of Classifiers via Generative Interpolation [56.620403243640396]
  Deep neural networks achieve superior performance for learning from independent and identically distributed (i.i.d.) data. However, their performance deteriorates significantly when handling out-of-distribution (OoD) data. We develop a simple yet effective method called Generative Interpolation to fuse generative models trained from multiple domains for synthesizing diverse OoD samples.
  arXiv  Detail & Related papers  (2023-07-23T03:53:53Z)
• T1: Scaling Diffusion Probabilistic Fields to High-Resolution on Unified Visual Modalities [69.16656086708291]
  Diffusion Probabilistic Field (DPF) models the distribution of continuous functions defined over metric spaces. We propose a new model comprising of a view-wise sampling algorithm to focus on local structure learning. The model can be scaled to generate high-resolution data while unifying multiple modalities.
  arXiv  Detail & Related papers  (2023-05-24T03:32:03Z)
• VTAE: Variational Transformer Autoencoder with Manifolds Learning [144.0546653941249]
  Deep generative models have demonstrated successful applications in learning non-linear data distributions through a number of latent variables. The nonlinearity of the generator implies that the latent space shows an unsatisfactory projection of the data space, which results in poor representation learning. We show that geodesics and accurate computation can substantially improve the performance of deep generative models.
  arXiv  Detail & Related papers  (2023-04-03T13:13:19Z)
• Semi-Supervised Manifold Learning with Complexity Decoupled Chart Autoencoders [45.29194877564103]
  This work introduces a chart autoencoder with an asymmetric encoding-decoding process that can incorporate additional semi-supervised information such as class labels. We discuss the approximation power of such networks and derive a bound that essentially depends on the intrinsic dimension of the data manifold rather than the dimension of ambient space.
  arXiv  Detail & Related papers  (2022-08-22T19:58:03Z)
• Adversarial Manifold Matching via Deep Metric Learning for Generative Modeling [5.5840609705075055]
  We propose a manifold matching approach to generative models which includes a distribution generator and a metric generator. The distribution generator aims at generating samples that follow some distribution condensed around the real data manifold. The metric generator utilizes both real data and generated samples to learn a distance metric.
  arXiv  Detail & Related papers  (2021-06-20T23:25:01Z)
• Flow-based Generative Models for Learning Manifold to Manifold Mappings [39.60406116984869]
  We introduce three kinds of invertible layers for manifold-valued data, which are analogous to their functionality in flow-based generative models. We show promising results where we can reliably and accurately reconstruct brain images of a field of orientation distribution functions.
  arXiv  Detail & Related papers  (2020-12-18T02:19:18Z)
• Variational Mixture of Normalizing Flows [0.0]
  Deep generative models, such as generative adversarial networks autociteGAN, variational autoencoders autocitevaepaper, and their variants, have seen wide adoption for the task of modelling complex data distributions. Normalizing flows have overcome this limitation by leveraging the change-of-suchs formula for probability density functions. The present work overcomes this by using normalizing flows as components in a mixture model and devising an end-to-end training procedure for such a model.
  arXiv  Detail & Related papers  (2020-09-01T17:20:08Z)
• Evaluating the Disentanglement of Deep Generative Models through Manifold Topology [66.06153115971732]
  We present a method for quantifying disentanglement that only uses the generative model. We empirically evaluate several state-of-the-art models across multiple datasets.
  arXiv  Detail & Related papers  (2020-06-05T20:54:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.