Diagonally-Weighted Generalized Method of Moments Estimation for Gaussian Mixture Modeling

• URL: http://arxiv.org/abs/2507.20459v1
• Date: Mon, 28 Jul 2025 01:24:55 GMT
• Title: Diagonally-Weighted Generalized Method of Moments Estimation for Gaussian Mixture Modeling
• Authors: Liu Zhang, Oscar Mickelin, Sheng Xu, Amit Singer,
• Abstract summary: The diagonally-weighted GMM (DGMM) achieves a balance among statistical efficiency, computational complexity, and numerical stability.<n>We apply DGMM to study the parameter estimation problem for weakly separated heteroscedastic low-rank Gaussian mixtures.<n>In numerical studies, DGMM attains smaller estimation errors while requiring substantially shorter runtimes than MM and GMM.
• Score: 7.266474956854856
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Since Pearson [Philosophical Transactions of the Royal Society of London. A, 185 (1894), pp. 71-110] first applied the method of moments (MM) for modeling data as a mixture of one-dimensional Gaussians, moment-based estimation methods have proliferated. Among these methods, the generalized method of moments (GMM) improves the statistical efficiency of MM by weighting the moments appropriately. However, the computational complexity and storage complexity of MM and GMM grow exponentially with the dimension, making these methods impractical for high-dimensional data or when higher-order moments are required. Such computational bottlenecks are more severe in GMM since it additionally requires estimating a large weighting matrix. To overcome these bottlenecks, we propose the diagonally-weighted GMM (DGMM), which achieves a balance among statistical efficiency, computational complexity, and numerical stability. We apply DGMM to study the parameter estimation problem for weakly separated heteroscedastic low-rank Gaussian mixtures and design a computationally efficient and numerically stable algorithm that obtains the DGMM estimator without explicitly computing or storing the moment tensors. We implement the proposed algorithm and empirically validate the advantages of DGMM: in numerical studies, DGMM attains smaller estimation errors while requiring substantially shorter runtime than MM and GMM. The code and data will be available upon publication at https://github.com/liu-lzhang/dgmm.

Related papers

• Sparse high-dimensional linear mixed modeling with a partitioned empirical Bayes ECM algorithm [41.25603565852633]
  This work presents an efficient and accurate Bayesian framework for high-dimensional LMMs. The novelty of the approach lies in its partitioning and parameter expansion as well as its fast and scalable computation. A real-world example is provided using data from a study of lupus in children, where we identify genes and clinical factors associated with a new lupus biomarker and predict the biomarker over time.
  arXiv  Detail & Related papers  (2023-10-18T19:34:56Z)
• Large-scale gradient-based training of Mixtures of Factor Analyzers [67.21722742907981]
  This article contributes both a theoretical analysis as well as a new method for efficient high-dimensional training by gradient descent. We prove that MFA training and inference/sampling can be performed based on precision matrices, which does not require matrix inversions after training is completed. Besides the theoretical analysis and matrices, we apply MFA to typical image datasets such as SVHN and MNIST, and demonstrate the ability to perform sample generation and outlier detection.
  arXiv  Detail & Related papers  (2023-08-26T06:12:33Z)
• SGMM: Stochastic Approximation to Generalized Method of Moments [8.48870560391056]
  We introduce a new class of algorithms, Generalized Method of Moments (SGMM) for estimation and inference on (overidentified) moment restriction models. Our SGMM is a novel approximation to the popular Hansen (1982) (offline) GMM, and offers fast and scalable implementation with the ability to handle streaming datasets in real time.
  arXiv  Detail & Related papers  (2023-08-25T00:22:45Z)
• Algorithme EM r\'egularis\'e [0.0]
  This paper presents a regularized version of the EM algorithm that efficiently uses prior knowledge to cope with a small sample size. Experiments on real data highlight the good performance of the proposed algorithm for clustering purposes.
  arXiv  Detail & Related papers  (2023-07-04T23:19:25Z)
• Regularized EM algorithm [9.367612782346205]
  We present a regularized EM algorithm for GMM-s that can make efficient use of such prior knowledge as well as cope with LSS situations. We show that the theoretical guarantees of convergence hold, leading to better performing EM algorithm for structured covariance matrix models or with low sample settings.
  arXiv  Detail & Related papers  (2023-03-27T08:32:20Z)
• Gaussian process regression and conditional Karhunen-Lo\'{e}ve models for data assimilation in inverse problems [68.8204255655161]
  We present a model inversion algorithm, CKLEMAP, for data assimilation and parameter estimation in partial differential equation models. The CKLEMAP method provides better scalability compared to the standard MAP method.
  arXiv  Detail & Related papers  (2023-01-26T18:14:12Z)
• Stochastic First-Order Learning for Large-Scale Flexibly Tied Gaussian Mixture Model [3.4546761246181696]
  We propose a new optimization algorithm on the manifold of Gaussian Mixture Models (GMMs) We observe that methods can outperform the expectation-maximization algorithm in terms of attaining better likelihood, needing fewer epochs for convergence, and consuming less time per each epoch.
  arXiv  Detail & Related papers  (2022-12-11T04:24:52Z)
• Cauchy-Schwarz Regularized Autoencoder [68.80569889599434]
  Variational autoencoders (VAE) are a powerful and widely-used class of generative models. We introduce a new constrained objective based on the Cauchy-Schwarz divergence, which can be computed analytically for GMMs. Our objective improves upon variational auto-encoding models in density estimation, unsupervised clustering, semi-supervised learning, and face analysis.
  arXiv  Detail & Related papers  (2021-01-06T17:36:26Z)
• Generalized Matrix Factorization: efficient algorithms for fitting generalized linear latent variable models to large data arrays [62.997667081978825]
  Generalized Linear Latent Variable models (GLLVMs) generalize such factor models to non-Gaussian responses. Current algorithms for estimating model parameters in GLLVMs require intensive computation and do not scale to large datasets. We propose a new approach for fitting GLLVMs to high-dimensional datasets, based on approximating the model using penalized quasi-likelihood.
  arXiv  Detail & Related papers  (2020-10-06T04:28:19Z)
• Coded Stochastic ADMM for Decentralized Consensus Optimization with Edge Computing [113.52575069030192]
  Big data, including applications with high security requirements, are often collected and stored on multiple heterogeneous devices, such as mobile devices, drones and vehicles. Due to the limitations of communication costs and security requirements, it is of paramount importance to extract information in a decentralized manner instead of aggregating data to a fusion center. We consider the problem of learning model parameters in a multi-agent system with data locally processed via distributed edge nodes. A class of mini-batch alternating direction method of multipliers (ADMM) algorithms is explored to develop the distributed learning model.
  arXiv  Detail & Related papers  (2020-10-02T10:41:59Z)
• Effective Learning of a GMRF Mixture Model [8.336315962271396]
  We propose restricting the GMM to a Gaussian Markov Random Field Mixture Model (GMRF-MM) When the sparsity pattern of each matrix is known, we propose an efficient optimization method for the Maximum Likelihood Estimate (MLE) of that matrix. We show that our "debiasing" approach outperforms GLASSO in both the single-GMRF and the GMRF-MM cases.
  arXiv  Detail & Related papers  (2020-05-18T19:00:14Z)
• Q-GADMM: Quantized Group ADMM for Communication Efficient Decentralized Machine Learning [66.18202188565922]
  We propose a communication-efficient decentralized machine learning (ML) algorithm, coined QGADMM (QGADMM)<n>We develop a novel quantization method to adaptively adjust modelization levels and their probabilities, while proving the convergence of QGADMM for convex functions.
  arXiv  Detail & Related papers  (2019-10-23T10:47:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.