Parsimonious Gaussian mixture models with piecewise-constant eigenvalue profiles

• URL: http://arxiv.org/abs/2507.01542v1
• Date: Wed, 02 Jul 2025 09:52:56 GMT
• Title: Parsimonious Gaussian mixture models with piecewise-constant eigenvalue profiles
• Authors: Tom Szwagier, Pierre-Alexandre Mattei, Charles Bouveyron, Xavier Pennec,
• Abstract summary: We introduce a new family of parsimonious GMMs with piecewise-constant covariance eigenvalue profiles.<n>We show the superior likelihood-parsimony tradeoffs achieved by our models on a variety of unsupervised experiments.
• Score: 16.798207551231872
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Gaussian mixture models (GMMs) are ubiquitous in statistical learning, particularly for unsupervised problems. While full GMMs suffer from the overparameterization of their covariance matrices in high-dimensional spaces, spherical GMMs (with isotropic covariance matrices) certainly lack flexibility to fit certain anisotropic distributions. Connecting these two extremes, we introduce a new family of parsimonious GMMs with piecewise-constant covariance eigenvalue profiles. These extend several low-rank models like the celebrated mixtures of probabilistic principal component analyzers (MPPCA), by enabling any possible sequence of eigenvalue multiplicities. If the latter are prespecified, then we can naturally derive an expectation-maximization (EM) algorithm to learn the mixture parameters. Otherwise, to address the notoriously-challenging issue of jointly learning the mixture parameters and hyperparameters, we propose a componentwise penalized EM algorithm, whose monotonicity is proven. We show the superior likelihood-parsimony tradeoffs achieved by our models on a variety of unsupervised experiments: density fitting, clustering and single-image denoising.

Related papers

• Toward Global Convergence of Gradient EM for Over-Parameterized Gaussian Mixture Models [47.294535652946095]
  We study the gradient Expectation-Maximization (EM) algorithm for Gaussian Mixture Models (GMM)<n>This is the first global convergence result for Gaussian mixtures with more than $2$ components.
  arXiv  Detail & Related papers  (2024-06-29T16:44:29Z)
• Adaptive Fuzzy C-Means with Graph Embedding [84.47075244116782]
  Fuzzy clustering algorithms can be roughly categorized into two main groups: Fuzzy C-Means (FCM) based methods and mixture model based methods. We propose a novel FCM based clustering model that is capable of automatically learning an appropriate membership degree hyper- parameter value.
  arXiv  Detail & Related papers  (2024-05-22T08:15:50Z)
• Robust scalable initialization for Bayesian variational inference with multi-modal Laplace approximations [0.0]
  Variational mixtures with full-covariance structures suffer from a quadratic growth due to variational parameters with the number of parameters. We propose a method for constructing an initial Gaussian model approximation that can be used to warm-start variational inference.
  arXiv  Detail & Related papers  (2023-07-12T19:30:04Z)
• On the properties of Gaussian Copula Mixture Models [0.0]
  The paper presents the mathematical definition of GCMM and explores the properties of its likelihood function. The paper proposes extended Expectation algorithms to estimate parameters for the mixture of copulas.
  arXiv  Detail & Related papers  (2023-05-02T14:59:37Z)
• On Learning Mixture Models with Sparse Parameters [44.3425205248937]
  We study mixtures with high dimensional sparse latent parameter vectors and consider the problem of support recovery of those vectors. We provide efficient algorithms for support recovery that have a logarithmic sample complexity dependence on the dimensionality of the latent space.
  arXiv  Detail & Related papers  (2022-02-24T07:44:23Z)
• A Robust and Flexible EM Algorithm for Mixtures of Elliptical Distributions with Missing Data [71.9573352891936]
  This paper tackles the problem of missing data imputation for noisy and non-Gaussian data. A new EM algorithm is investigated for mixtures of elliptical distributions with the property of handling potential missing data. Experimental results on synthetic data demonstrate that the proposed algorithm is robust to outliers and can be used with non-Gaussian data.
  arXiv  Detail & Related papers  (2022-01-28T10:01:37Z)
• Learning Gaussian Mixtures with Generalised Linear Models: Precise Asymptotics in High-dimensions [79.35722941720734]
  Generalised linear models for multi-class classification problems are one of the fundamental building blocks of modern machine learning tasks. We prove exacts characterising the estimator in high-dimensions via empirical risk minimisation. We discuss how our theory can be applied beyond the scope of synthetic data.
  arXiv  Detail & Related papers  (2021-06-07T16:53:56Z)
• A Rigorous Link Between Self-Organizing Maps and Gaussian Mixture Models [78.6363825307044]
  This work presents a mathematical treatment of the relation between Self-Organizing Maps (SOMs) and Gaussian Mixture Models (GMMs) We show that energy-based SOM models can be interpreted as performing gradient descent. This link allows to treat SOMs as generative probabilistic models, giving a formal justification for using SOMs to detect outliers, or for sampling.
  arXiv  Detail & Related papers  (2020-09-24T14:09:04Z)
• Clustering of non-Gaussian data by variational Bayes for normal inverse Gaussian mixture models [0.0]
  In practical situations, there are many non-Gaussian data that are heavy-tailed and/or asymmetric. For NIG mixture models, both expectation-maximization method and variational Bayesian (VB) algorithms have been proposed. We propose another VB algorithm for NIG mixture that improves on the shortcomings. We also propose an extension of Dirichlet process mixture models to overcome the difficulty in determining the number of clusters.
  arXiv  Detail & Related papers  (2020-09-13T14:13:27Z)
• Self-regularizing Property of Nonparametric Maximum Likelihood Estimator in Mixture Models [39.27013036481509]
  We introduce the nonparametric maximum likelihood (NPMLE) model for general Gaussian mixtures. We show that with high probability the NPMLE based on a sample size has $O(log n)$ atoms (mass points) Notably, any mixture is statistically in from a finite one with $Olog selection.
  arXiv  Detail & Related papers  (2020-08-19T03:39:13Z)
• Algebraic and Analytic Approaches for Parameter Learning in Mixture Models [66.96778152993858]
  We present two different approaches for parameter learning in several mixture models in one dimension. For some of these distributions, our results represent the first guarantees for parameter estimation.
  arXiv  Detail & Related papers  (2020-01-19T05:10:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.