On the Optimization Landscape of Maximum Mean Discrepancy

• URL: http://arxiv.org/abs/2110.13452v2
• Date: Fri, 3 May 2024 19:41:06 GMT
• Title: On the Optimization Landscape of Maximum Mean Discrepancy
• Authors: Itai Alon, Amir Globerson, Ami Wiesel,
• Abstract summary: Generative models have been successfully used for generating realistic signals. Because the likelihood function is typically intractable in most of these models, the common practice is to "implicit" that avoid likelihood calculation. In particular, it is not understood when they can minimize their non-repancy objectives globally.
• Score: 26.661542645011046
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Generative models have been successfully used for generating realistic signals. Because the likelihood function is typically intractable in most of these models, the common practice is to use "implicit" models that avoid likelihood calculation. However, it is hard to obtain theoretical guarantees for such models. In particular, it is not understood when they can globally optimize their non-convex objectives. Here we provide such an analysis for the case of Maximum Mean Discrepancy (MMD) learning of generative models. We prove several optimality results, including for a Gaussian distribution with low rank covariance (where likelihood is inapplicable) and a mixture of Gaussians. Our analysis shows that that the MMD optimization landscape is benign in these cases, and therefore gradient based methods will globally minimize the MMD objective.

Related papers

• A Gradient Analysis Framework for Rewarding Good and Penalizing Bad Examples in Language Models [63.949883238901414]
  We present a unique angle of gradient analysis of loss functions that simultaneously reward good examples and penalize bad ones in LMs. We find that ExMATE serves as a superior surrogate for MLE, and that combining DPO with ExMATE instead of MLE further enhances both the statistical (5-7%) and generative (+18% win rate) performance.
  arXiv  Detail & Related papers  (2024-08-29T17:46:18Z)
• Polynomial Chaos Expanded Gaussian Process [2.287415292857564]
  In complex and unknown processes, global models are initially generated over the entire experimental space. This study addresses the need for models that effectively represent both global and local experimental spaces.
  arXiv  Detail & Related papers  (2024-05-02T07:11:05Z)
• Soft Preference Optimization: Aligning Language Models to Expert Distributions [40.84391304598521]
  SPO is a method for aligning generative models, such as Large Language Models (LLMs), with human preferences. SPO integrates preference loss with a regularization term across the model's entire output distribution. We showcase SPO's methodology, its theoretical foundation, and its comparative advantages in simplicity, computational efficiency, and alignment precision.
  arXiv  Detail & Related papers  (2024-04-30T19:48:55Z)
• Probabilistic Unrolling: Scalable, Inverse-Free Maximum Likelihood Estimation for Latent Gaussian Models [69.22568644711113]
  We introduce probabilistic unrolling, a method that combines Monte Carlo sampling with iterative linear solvers to circumvent matrix inversions. Our theoretical analyses reveal that unrolling and backpropagation through the iterations of the solver can accelerate gradient estimation for maximum likelihood estimation. In experiments on simulated and real data, we demonstrate that probabilistic unrolling learns latent Gaussian models up to an order of magnitude faster than gradient EM, with minimal losses in model performance.
  arXiv  Detail & Related papers  (2023-06-05T21:08:34Z)
• Maximum Likelihood Estimation in Gaussian Process Regression is Ill-Posed [7.018149356115115]
  It remains an open problem to establish the circumstances in which maximum likelihood estimation is well-posed. This article identifies scenarios where the maximum likelihood estimator fails to be well-posed. Although the failure of maximum likelihood estimation is part of Gaussian process folklore, these rigorous theoretical results appear to be the first of their kind.
  arXiv  Detail & Related papers  (2022-03-17T09:00:39Z)
• Learning to Estimate Without Bias [57.82628598276623]
  Gauss theorem states that the weighted least squares estimator is a linear minimum variance unbiased estimation (MVUE) in linear models. In this paper, we take a first step towards extending this result to non linear settings via deep learning with bias constraints. A second motivation to BCE is in applications where multiple estimates of the same unknown are averaged for improved performance.
  arXiv  Detail & Related papers  (2021-10-24T10:23:51Z)
• Latent Gaussian Model Boosting [0.0]
  Tree-boosting shows excellent predictive accuracy on many data sets. We obtain increased predictive accuracy compared to existing approaches in both simulated and real-world data experiments.
  arXiv  Detail & Related papers  (2021-05-19T07:36:30Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• Goal-directed Generation of Discrete Structures with Conditional Generative Models [85.51463588099556]
  We introduce a novel approach to directly optimize a reinforcement learning objective, maximizing an expected reward. We test our methodology on two tasks: generating molecules with user-defined properties and identifying short python expressions which evaluate to a given target value.
  arXiv  Detail & Related papers  (2020-10-05T20:03:13Z)
• Likelihood-Free Inference with Deep Gaussian Processes [70.74203794847344]
  Surrogate models have been successfully used in likelihood-free inference to decrease the number of simulator evaluations. We propose a Deep Gaussian Process (DGP) surrogate model that can handle more irregularly behaved target distributions. Our experiments show how DGPs can outperform GPs on objective functions with multimodal distributions and maintain a comparable performance in unimodal cases.
  arXiv  Detail & Related papers  (2020-06-18T14:24:05Z)
• Expected Information Maximization: Using the I-Projection for Mixture Density Estimation [22.096148237257644]
  Modelling highly multi-modal data is a challenging problem in machine learning. We present a new algorithm called Expected Information Maximization (EIM) for computing the I-projection. We show that our algorithm is much more effective in computing the I-projection than recent GAN approaches.
  arXiv  Detail & Related papers  (2020-01-23T17:24:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.