Smoothness and Stability in GANs

• URL: http://arxiv.org/abs/2002.04185v1
• Date: Tue, 11 Feb 2020 03:08:28 GMT
• Title: Smoothness and Stability in GANs
• Authors: Casey Chu, Kentaro Minami, Kenji Fukumizu
• Abstract summary: Generative adversarial networks, or GANs, commonly display unstable behavior during training. We develop a principled theoretical framework for understanding the stability of various types of GANs.
• Score: 21.01604897837572
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative adversarial networks, or GANs, commonly display unstable behavior during training. In this work, we develop a principled theoretical framework for understanding the stability of various types of GANs. In particular, we derive conditions that guarantee eventual stationarity of the generator when it is trained with gradient descent, conditions that must be satisfied by the divergence that is minimized by the GAN and the generator's architecture. We find that existing GAN variants satisfy some, but not all, of these conditions. Using tools from convex analysis, optimal transport, and reproducing kernels, we construct a GAN that fulfills these conditions simultaneously. In the process, we explain and clarify the need for various existing GAN stabilization techniques, including Lipschitz constraints, gradient penalties, and smooth activation functions.

Related papers

• A New Formulation of Lipschitz Constrained With Functional Gradient Learning for GANs [52.55025869932486]
  This paper introduces a promising alternative method for training Generative Adversarial Networks (GANs) on large-scale datasets with clear theoretical guarantees. We propose a novel Lipschitz-constrained Functional Gradient GANs learning (Li-CFG) method to stabilize the training of GAN. We demonstrate that the neighborhood size of the latent vector can be reduced by increasing the norm of the discriminator gradient.
  arXiv  Detail & Related papers  (2025-01-20T02:48:07Z)
• Unified theoretical guarantees for stability, consistency, and convergence in neural PDE solvers from non-IID data to physics-informed networks [0.0]
  We establish a unified theoretical framework addressing the stability, consistency, and convergence of neural networks under realistic training conditions.<n>For standard supervised learning with dependent data, we derive uniform stability bounds for gradient-based methods.<n>In federated learning with heterogeneous data, we quantify model inconsistency via curvature-aware aggregation and information-theoretic divergence.
  arXiv  Detail & Related papers  (2024-09-08T08:48:42Z)
• Convergences for Minimax Optimization Problems over Infinite-Dimensional Spaces Towards Stability in Adversarial Training [0.6008132390640294]
  Training neural networks that require adversarial optimization, such as generative adversarial networks (GANs), suffers from instability. In this study, we tackle this problem theoretically through a functional analysis.
  arXiv  Detail & Related papers  (2023-12-02T01:15:57Z)
• Generator Identification for Linear SDEs with Additive and Multiplicative Noise [48.437815378088466]
  identifiability conditions are crucial in causal inference using linear SDEs. We derive a sufficient and necessary condition for identifying the generator of linear SDEs with additive noise. We offer geometric interpretations of the derived identifiability conditions to enhance their understanding.
  arXiv  Detail & Related papers  (2023-10-30T12:28:53Z)
• Numerically Stable Sparse Gaussian Processes via Minimum Separation using Cover Trees [57.67528738886731]
  We study the numerical stability of scalable sparse approximations based on inducing points. For low-dimensional tasks such as geospatial modeling, we propose an automated method for computing inducing points satisfying these conditions.
  arXiv  Detail & Related papers  (2022-10-14T15:20:17Z)
• Dynamics of Fourier Modes in Torus Generative Adversarial Networks [0.8189696720657245]
  Generative Adversarial Networks (GANs) are powerful Machine Learning models capable of generating fully synthetic samples of a desired phenomenon with a high resolution. Despite their success, the training process of a GAN is highly unstable and typically it is necessary to implement several accessory perturbations to the networks to reach an acceptable convergence of the model. We introduce a novel method to analyze the convergence and stability in the training of Generative Adversarial Networks.
  arXiv  Detail & Related papers  (2022-09-05T09:03:22Z)
• Revisiting GANs by Best-Response Constraint: Perspective, Methodology, and Application [49.66088514485446]
  Best-Response Constraint (BRC) is a general learning framework to explicitly formulate the potential dependency of the generator on the discriminator. We show that even with different motivations and formulations, a variety of existing GANs ALL can be uniformly improved by our flexible BRC methodology.
  arXiv  Detail & Related papers  (2022-05-20T12:42:41Z)
• Training Generative Adversarial Networks by Solving Ordinary Differential Equations [54.23691425062034]
  We study the continuous-time dynamics induced by GAN training. From this perspective, we hypothesise that instabilities in training GANs arise from the integration error. We experimentally verify that well-known ODE solvers (such as Runge-Kutta) can stabilise training.
  arXiv  Detail & Related papers  (2020-10-28T15:23:49Z)
• Conditional Hybrid GAN for Sequence Generation [56.67961004064029]
  We propose a novel conditional hybrid GAN (C-Hybrid-GAN) to solve this issue. We exploit the Gumbel-Softmax technique to approximate the distribution of discrete-valued sequences. We demonstrate that the proposed C-Hybrid-GAN outperforms the existing methods in context-conditioned discrete-valued sequence generation.
  arXiv  Detail & Related papers  (2020-09-18T03:52:55Z)
• SGD for Structured Nonconvex Functions: Learning Rates, Minibatching and Interpolation [17.199023009789308]
  The Expected assumption of SGD (SGD) is being used routinely for non-artisan functions. In this paper, we show a paradigms for convergence to a smooth non-linear setting. We also provide theoretical guarantees for different step-size conditions.
  arXiv  Detail & Related papers  (2020-06-18T07:05:56Z)
• Fine-Grained Analysis of Stability and Generalization for Stochastic Gradient Descent [55.85456985750134]
  We introduce a new stability measure called on-average model stability, for which we develop novel bounds controlled by the risks of SGD iterates. This yields generalization bounds depending on the behavior of the best model, and leads to the first-ever-known fast bounds in the low-noise setting. To our best knowledge, this gives the firstever-known stability and generalization for SGD with even non-differentiable loss functions.
  arXiv  Detail & Related papers  (2020-06-15T06:30:19Z)
• Cumulant GAN [17.4556035872983]
  We propose a novel loss function for training Generative Adversarial Networks (GANs) We show that the corresponding optimization problem is equivalent to R'enyi divergence minimization. We experimentally demonstrate that image generation is more robust relative to Wasserstein GAN.
  arXiv  Detail & Related papers  (2020-06-11T17:23:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.