GANs Secretly Perform Approximate Bayesian Model Selection

• URL: http://arxiv.org/abs/2507.00651v1
• Date: Tue, 01 Jul 2025 10:49:06 GMT
• Title: GANs Secretly Perform Approximate Bayesian Model Selection
• Authors: Maurizio Filippone, Marius P. Linhard,
• Abstract summary: Generative Adversarial Networks (GANs) are popular and successful generative models.<n>We explain the success and limitations of GANs by interpreting them as probabilistic generative models.
• Score: 9.901238730488657
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Generative Adversarial Networks (GANs) are popular and successful generative models. Despite their success, optimization is notoriously challenging and they require regularization against overfitting. In this work, we explain the success and limitations of GANs by interpreting them as probabilistic generative models. This interpretation enables us to view GANs as Bayesian neural networks with partial stochasticity, allowing us to establish conditions of universal approximation. We can then cast the adversarial-style optimization of several variants of GANs as the optimization of a proxy for the marginal likelihood. Taking advantage of the connection between marginal likelihood optimization and Occam's razor, we can define regularization and optimization strategies to smooth the loss landscape and search for solutions with minimum description length, which are associated with flat minima and good generalization. The results on a wide range of experiments indicate that these strategies lead to performance improvements and pave the way to a deeper understanding of regularization strategies for GANs.

Related papers

• Optimizers Qualitatively Alter Solutions And We Should Leverage This [62.662640460717476]
  Deep Neural Networks (DNNs) can not guarantee convergence to a unique global minimum of the loss when using only local information, such as SGD.<n>We argue that the community should aim at understanding the biases of already existing methods, as well as aim to build new DNNs with the explicit intent of inducing certain properties of the solution.
  arXiv  Detail & Related papers  (2025-07-16T13:33:31Z)
• Indirect Query Bayesian Optimization with Integrated Feedback [17.66813850517961]
  We develop a new class of Bayesian optimization problems where integrated feedback is given via a conditional expectation of the unknown function $f$ to be optimized.<n>The goal is to find the global optimum of $f$ by adaptively querying and observing in the space transformed by the conditional distribution.<n>This is motivated by real-world applications where one cannot access direct feedback due to privacy, hardware or computational constraints.
  arXiv  Detail & Related papers  (2024-12-18T07:20:33Z)
• Diffusion Models as Network Optimizers: Explorations and Analysis [71.69869025878856]
  generative diffusion models (GDMs) have emerged as a promising new approach to network optimization.<n>In this study, we first explore the intrinsic characteristics of generative models.<n>We provide a concise theoretical and intuitive demonstration of the advantages of generative models over discriminative network optimization.
  arXiv  Detail & Related papers  (2024-11-01T09:05:47Z)
• DiffSG: A Generative Solver for Network Optimization with Diffusion Model [75.27274046562806]
  Generative diffusion models are popular in various cross-domain applications.<n>These models hold promise in tackling complex network optimization problems.<n>We propose a new framework for generative diffusion models called Diffusion Model-based Solution Generation.
  arXiv  Detail & Related papers  (2024-08-13T07:56:21Z)
• Discovering Preference Optimization Algorithms with and for Large Language Models [50.843710797024805]
  offline preference optimization is a key method for enhancing and controlling the quality of Large Language Model (LLM) outputs. We perform objective discovery to automatically discover new state-of-the-art preference optimization algorithms without (expert) human intervention. Experiments demonstrate the state-of-the-art performance of DiscoPOP, a novel algorithm that adaptively blends logistic and exponential losses.
  arXiv  Detail & Related papers  (2024-06-12T16:58:41Z)
• Reducing the Need for Backpropagation and Discovering Better Optima With Explicit Optimizations of Neural Networks [4.807347156077897]
  We propose a computationally efficient alternative for optimizing neural networks. We derive an explicit solution to a simple feed-forward language model. We show that explicit solutions perform near-optimality in experiments.
  arXiv  Detail & Related papers  (2023-11-13T17:38:07Z)
• G-TRACER: Expected Sharpness Optimization [1.2183405753834562]
  G-TRACER promotes generalization by seeking flat minima, and has a sound theoretical basis as an approximation to a natural-gradient descent based optimization of a generalized Bayes objective. We show that the method converges to a neighborhood of a local minimum of the unregularized objective, and demonstrate competitive performance on a number of benchmark computer vision and NLP datasets.
  arXiv  Detail & Related papers  (2023-06-24T09:28:49Z)
• Backpropagation of Unrolled Solvers with Folded Optimization [55.04219793298687]
  The integration of constrained optimization models as components in deep networks has led to promising advances on many specialized learning tasks. One typical strategy is algorithm unrolling, which relies on automatic differentiation through the operations of an iterative solver. This paper provides theoretical insights into the backward pass of unrolled optimization, leading to a system for generating efficiently solvable analytical models of backpropagation.
  arXiv  Detail & Related papers  (2023-01-28T01:50:42Z)
• Revisiting GANs by Best-Response Constraint: Perspective, Methodology, and Application [49.66088514485446]
  Best-Response Constraint (BRC) is a general learning framework to explicitly formulate the potential dependency of the generator on the discriminator. We show that even with different motivations and formulations, a variety of existing GANs ALL can be uniformly improved by our flexible BRC methodology.
  arXiv  Detail & Related papers  (2022-05-20T12:42:41Z)
• AGGLIO: Global Optimization for Locally Convex Functions [5.221860952360943]
  This paper presents AGG (Accelerated Optimization Generalized LInear-model) a stage-wise, global technique that offers provable convergence problems. AGG can be readily implemented using point as A-batch SGD updates and offers provable convergence as well as convergent experiments.
  arXiv  Detail & Related papers  (2021-11-06T18:15:56Z)
• Prb-GAN: A Probabilistic Framework for GAN Modelling [20.181803514993778]
  We present a new variation that uses dropout to create a distribution over the network parameters with the posterior learnt using variational inference. Our methods are extremely simple and require very little modification to existing GAN architecture.
  arXiv  Detail & Related papers  (2021-07-12T08:04:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.