Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence

• URL: http://arxiv.org/abs/2606.19876v1
• Date: Thu, 18 Jun 2026 07:34:33 GMT
• Title: Global Convergence of Gradient Descent for Score Matching in Gaussian Mixtures via Reverse Fisher Divergence
• Abstract summary: We study an alternative objective: the reverse Fisher divergence, where the expectation is taken with respect to the student distribution.<n>We prove global convergence guarantees under a $widetilde(1)$-separation assumption on the target means.<n>Our rely on a new Lyapunov-based analysis of the gradient descent dynamics, showing that the reverse Fisher divergence has a much more favorable optimization landscape than the forward Fisher divergence.
• Score: 67.12978375116599
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The score matching problem is a central training objective in modern generative modeling, diffusion models, fitting unnormalized statistical models, and inverse problems. A standard approach is to minimize the forward Fisher divergence, where the expectation is taken with respect to the teacher distribution. However, recent results show that even in simple Gaussian mixture model settings, this objective can lead to undesirable and initialization-dependent convergence behavior. In this paper, we study an alternative objective: the reverse Fisher divergence, where the expectation is taken with respect to the student distribution. We analyze gradient descent (GD) for fitting Gaussian mixture models and show that this change in the objective leads to significantly better optimization properties. First, when the teacher distribution is a single Gaussian and the student is a Gaussian mixture model with fixed weights and identity covariances, we prove the global convergence of GD from arbitrary initializations. Second, we extend the analysis to the case where the teacher is also a Gaussian mixture model and prove global convergence guarantees under a global random initialization scheme and a $\widetildeΩ(1)$-separation assumption on the target means. In particular, with high probability, each student component converges near its closest teacher component, and we provide conditions under which the student distribution converges in total variation distance. Our proofs rely on a new Lyapunov-based analysis of the gradient descent dynamics, showing that the reverse Fisher divergence has a much more favorable optimization landscape than the forward Fisher divergence.

Related papers

• Learning Probabilistic Filters with Strictly Proper Scoring Rules [8.61813579171868]
  We introduce the proper scoring ensemble filter (PSEF), an ensemble data assimilation method based on training an analysis map to approximate the filtering distribution.<n>We prove that, under a realizability assumption, the population objective is minimized by the true Bayesian filtering distribution.<n> Numerical experiments show that the learned filter accurately approximates challenging filtering distributions, including nonlinear, non-Gaussian, and multi-modal posteriors.
  arXiv  Detail & Related papers  (2026-06-25T01:04:21Z)
• A Unified View of Drifting and Score-Based Models [92.51878018325031]
  We make our relationship to the score-matching principle behind diffusion models precise by showing that drifting admits a score-based formulation on kernel-smoothed distributions.<n>We derive an exact decomposition for general radial kernels, and for the Laplace kernel we prove rigorous error bounds showing that drifting remains an accurate proxy for score matching in low-temperature and high-dimensional regimes.
  arXiv  Detail & Related papers  (2026-03-08T07:41:36Z)
• Convergence Dynamics of Over-Parameterized Score Matching for a Single Gaussian [48.340460104014]
  We study gradient descent for training models to learn a single Gaussian distribution.<n>We prove a global convergence result for gradient descent under multiple regimes.<n>This is the first work to establish global convergence guarantees for Gaussian mixtures with at least three components under the score matching framework.
  arXiv  Detail & Related papers  (2025-11-27T03:41:48Z)
• End-To-End Learning of Gaussian Mixture Priors for Diffusion Sampler [15.372235873766812]
  Learnable mixture priors offer improved control over exploration, adaptability to target support, and increased to counteract mode collapse.<n>Our experimental results demonstrate significant performance improvements across a diverse range of real-world and synthetic benchmark problems.
  arXiv  Detail & Related papers  (2025-03-01T14:58:14Z)
• Theory on Score-Mismatched Diffusion Models and Zero-Shot Conditional Samplers [49.97755400231656]
  We present the first performance guarantee with explicit dimensional dependencies for general score-mismatched diffusion samplers.<n>We show that score mismatches result in an distributional bias between the target and sampling distributions, proportional to the accumulated mismatch between the target and training distributions.<n>This result can be directly applied to zero-shot conditional samplers for any conditional model, irrespective of measurement noise.
  arXiv  Detail & Related papers  (2024-10-17T16:42:12Z)
• MGF: Mixed Gaussian Flow for Diverse Trajectory Prediction [72.70572835589158]
  We propose constructing a mixed Gaussian prior for a normalizing flow model for trajectory prediction.<n>Our method achieves state-of-the-art performance in the evaluation of both trajectory alignment and diversity on the popular UCY/ETH and SDD datasets.
  arXiv  Detail & Related papers  (2024-02-19T15:48:55Z)
• Robust scalable initialization for Bayesian variational inference with multi-modal Laplace approximations [0.0]
  Variational mixtures with full-covariance structures suffer from a quadratic growth due to variational parameters with the number of parameters. We propose a method for constructing an initial Gaussian model approximation that can be used to warm-start variational inference.
  arXiv  Detail & Related papers  (2023-07-12T19:30:04Z)
• Loss function based second-order Jensen inequality and its application to particle variational inference [112.58907653042317]
  Particle variational inference (PVI) uses an ensemble of models as an empirical approximation for the posterior distribution. PVI iteratively updates each model with a repulsion force to ensure the diversity of the optimized models. We derive a novel generalization error bound and show that it can be reduced by enhancing the diversity of models.
  arXiv  Detail & Related papers  (2021-06-09T12:13:51Z)
• Understanding Double Descent Requires a Fine-Grained Bias-Variance Decomposition [34.235007566913396]
  We describe an interpretable, symmetric decomposition of the variance into terms associated with the labels. We find that the bias decreases monotonically with the network width, but the variance terms exhibit non-monotonic behavior. We also analyze the strikingly rich phenomenology that arises.
  arXiv  Detail & Related papers  (2020-11-04T21:04:02Z)
• Cumulant GAN [17.4556035872983]
  We propose a novel loss function for training Generative Adversarial Networks (GANs) We show that the corresponding optimization problem is equivalent to R'enyi divergence minimization. We experimentally demonstrate that image generation is more robust relative to Wasserstein GAN.
  arXiv  Detail & Related papers  (2020-06-11T17:23:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.