Coupled regularized sample covariance matrix estimator for multiple classes

• URL: http://arxiv.org/abs/2011.04315v2
• Date: Tue, 9 Nov 2021 08:55:59 GMT
• Title: Coupled regularized sample covariance matrix estimator for multiple classes
• Authors: Elias Raninen and Esa Ollila
• Abstract summary: We consider regularized SCM (RSCM) estimators for multiclass problems that couple together two different target matrices for regularization. We derive the MSE optimal tuning parameters for the estimators as well as propose a method for their estimation. The MSE performance of the proposed coupled RSCMs are evaluated with simulations and in a regularized discriminant analysis (RDA) classification set-up on real data.
• Score: 14.41703014203756
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The estimation of covariance matrices of multiple classes with limited training data is a difficult problem. The sample covariance matrix (SCM) is known to perform poorly when the number of variables is large compared to the available number of samples. In order to reduce the mean squared error (MSE) of the SCM, regularized (shrinkage) SCM estimators are often used. In this work, we consider regularized SCM (RSCM) estimators for multiclass problems that couple together two different target matrices for regularization: the pooled (average) SCM of the classes and the scaled identity matrix. Regularization toward the pooled SCM is beneficial when the population covariances are similar, whereas regularization toward the identity matrix guarantees that the estimators are positive definite. We derive the MSE optimal tuning parameters for the estimators as well as propose a method for their estimation under the assumption that the class populations follow (unspecified) elliptical distributions with finite fourth-order moments. The MSE performance of the proposed coupled RSCMs are evaluated with simulations and in a regularized discriminant analysis (RDA) classification set-up on real data. The results based on three different real data sets indicate comparable performance to cross-validation but with a significant speed-up in computation time.

Related papers

• Selecting the Number of Communities for Weighted Degree-Corrected Stochastic Block Models [5.117940794592611]
  We investigate how to select the number of communities for weighted networks without a full likelihood modeling. We propose a novel weighted degree-corrected block model (DCSBM), in which the mean adjacency matrix is modeled as the same as in standard DCSBM. Our method of selecting the number of communities is based on a sequential testing framework, and in each step the weighted DCSBM is fitted via some spectral clustering method.
  arXiv  Detail & Related papers  (2024-06-08T03:47:38Z)
• Regularized Linear Discriminant Analysis Using a Nonlinear Covariance Matrix Estimator [11.887333567383239]
  Linear discriminant analysis (LDA) is a widely used technique for data classification. LDA becomes inefficient when the data covariance matrix is ill-conditioned. Regularized LDA methods have been proposed to cope with such a situation.
  arXiv  Detail & Related papers  (2024-01-31T11:37:14Z)
• DCID: Deep Canonical Information Decomposition [84.59396326810085]
  We consider the problem of identifying the signal shared between two one-dimensional target variables. We propose ICM, an evaluation metric which can be used in the presence of ground-truth labels. We also propose Deep Canonical Information Decomposition (DCID) - a simple, yet effective approach for learning the shared variables.
  arXiv  Detail & Related papers  (2023-06-27T16:59:06Z)
• Regularized EM algorithm [9.367612782346205]
  We present a regularized EM algorithm for GMM-s that can make efficient use of such prior knowledge as well as cope with LSS situations. We show that the theoretical guarantees of convergence hold, leading to better performing EM algorithm for structured covariance matrix models or with low sample settings.
  arXiv  Detail & Related papers  (2023-03-27T08:32:20Z)
• Compound Batch Normalization for Long-tailed Image Classification [77.42829178064807]
  We propose a compound batch normalization method based on a Gaussian mixture. It can model the feature space more comprehensively and reduce the dominance of head classes. The proposed method outperforms existing methods on long-tailed image classification.
  arXiv  Detail & Related papers  (2022-12-02T07:31:39Z)
• CARMS: Categorical-Antithetic-REINFORCE Multi-Sample Gradient Estimator [60.799183326613395]
  We propose an unbiased estimator for categorical random variables based on multiple mutually negatively correlated (jointly antithetic) samples. CARMS combines REINFORCE with copula based sampling to avoid duplicate samples and reduce its variance, while keeping the estimator unbiased using importance sampling. We evaluate CARMS on several benchmark datasets on a generative modeling task, as well as a structured output prediction task, and find it to outperform competing methods including a strong self-control baseline.
  arXiv  Detail & Related papers  (2021-10-26T20:14:30Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Benchmarking Multivariate Time Series Classification Algorithms [69.12151492736524]
  Time Series Classification (TSC) involved building predictive models for a discrete target variable from ordered, real valued, attributes. Over recent years, a new set of TSC algorithms have been developed which have made significant improvement over the previous state of the art. We review recently proposed bespoke MTSC algorithms based on deep learning, shapelets and bag of words approaches.
  arXiv  Detail & Related papers  (2020-07-26T15:56:40Z)
• High-Dimensional Quadratic Discriminant Analysis under Spiked Covariance Model [101.74172837046382]
  We propose a novel quadratic classification technique, the parameters of which are chosen such that the fisher-discriminant ratio is maximized. Numerical simulations show that the proposed classifier not only outperforms the classical R-QDA for both synthetic and real data but also requires lower computational complexity.
  arXiv  Detail & Related papers  (2020-06-25T12:00:26Z)
• Covariance Estimation for Matrix-valued Data [9.739753590548796]
  We propose a class of distribution-free regularized covariance estimation methods for high-dimensional matrix data. We formulate a unified framework for estimating bandable covariance, and introduce an efficient algorithm based on rank one unconstrained Kronecker product approximation. We demonstrate the superior finite-sample performance of our methods using simulations and real applications from a gridded temperature anomalies dataset and a S&P 500 stock data analysis.
  arXiv  Detail & Related papers  (2020-04-11T02:15:26Z)
• Sparse Covariance Estimation in Logit Mixture Models [0.0]
  This paper introduces a new data-driven methodology for estimating sparse covariance matrices of the random coefficients in logit mixture models. Our objective is to find optimal subsets of correlated coefficients for which we estimate covariances.
  arXiv  Detail & Related papers  (2020-01-14T20:19:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.