Regularisation for PCA- and SVD-type matrix factorisations

• URL: http://arxiv.org/abs/2106.12955v1
• Date: Thu, 24 Jun 2021 12:25:12 GMT
• Title: Regularisation for PCA- and SVD-type matrix factorisations
• Authors: Abdolrahman Khoshrou, Eric J. Pauwels
• Abstract summary: Singular Value Decomposition (SVD) and its close relative, Principal Component Analysis (PCA) are well-known linear matrix decomposition techniques. In this paper, we take another look at the problem of regularisation and show that different formulations of the minimisation problem lead to qualitatively different solutions.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Singular Value Decomposition (SVD) and its close relative, Principal Component Analysis (PCA), are well-known linear matrix decomposition techniques that are widely used in applications such as dimension reduction and clustering. However, an important limitation of SVD/PCA is its sensitivity to noise in the input data. In this paper, we take another look at the problem of regularisation and show that different formulations of the minimisation problem lead to qualitatively different solutions.

Related papers

• Synergistic eigenanalysis of covariance and Hessian matrices for enhanced binary classification [72.77513633290056]
  We present a novel approach that combines the eigenanalysis of a covariance matrix evaluated on a training set with a Hessian matrix evaluated on a deep learning model. Our method captures intricate patterns and relationships, enhancing classification performance.
  arXiv  Detail & Related papers  (2024-02-14T16:10:42Z)
• The Inductive Bias of Flatness Regularization for Deep Matrix Factorization [58.851514333119255]
  This work takes the first step toward understanding the inductive bias of the minimum trace of the Hessian solutions in deep linear networks. We show that for all depth greater than one, with the standard Isometry Property (RIP) on the measurements, minimizing the trace of Hessian is approximately equivalent to minimizing the Schatten 1-norm of the corresponding end-to-end matrix parameters.
  arXiv  Detail & Related papers  (2023-06-22T23:14:57Z)
• Numerical Optimizations for Weighted Low-rank Estimation on Language Model [73.12941276331316]
  Singular value decomposition (SVD) is one of the most popular compression methods that approximates a target matrix with smaller matrices. Standard SVD treats the parameters within the matrix with equal importance, which is a simple but unrealistic assumption. We show that our method can perform better than current SOTA methods in neural-based language models.
  arXiv  Detail & Related papers  (2022-11-02T00:58:02Z)
• Robust Singular Values based on L1-norm PCA [11.706222361809374]
  Singular-Value Decomposition (SVD) is a ubiquitous data analysis method in engineering, science, and statistics. We present a novel robust non-parametric method for SVD and singular-value estimation based on a L1-norm (sum of absolute values) formulation.
  arXiv  Detail & Related papers  (2022-10-21T16:42:49Z)
• Supervised Linear Dimension-Reduction Methods: Review, Extensions, and Comparisons [6.71092092685492]
  Principal component analysis (PCA) is a well-known linear dimension-reduction method that has been widely used in data analysis and modeling. This paper reviews selected techniques, extends some of them, and compares their performance through simulations. Two of these techniques, partial least squares (PLS) and least-squares PCA (LSPCA), consistently outperform the others in this study.
  arXiv  Detail & Related papers  (2021-09-09T17:57:25Z)
• Benign Overfitting of Constant-Stepsize SGD for Linear Regression [122.70478935214128]
  inductive biases are central in preventing overfitting empirically. This work considers this issue in arguably the most basic setting: constant-stepsize SGD for linear regression. We reflect on a number of notable differences between the algorithmic regularization afforded by (unregularized) SGD in comparison to ordinary least squares.
  arXiv  Detail & Related papers  (2021-03-23T17:15:53Z)
• Reduction of the Number of Variables in Parametric Constrained Least-Squares Problems [0.20305676256390928]
  This paper proposes techniques for reducing the number of involved optimization variables. We show the good performance of the proposed techniques in numerical tests and in a linearized MPC problem of a nonlinear benchmark process.
  arXiv  Detail & Related papers  (2020-12-18T18:26:40Z)
• Robust Low-rank Matrix Completion via an Alternating Manifold Proximal Gradient Continuation Method [47.80060761046752]
  Robust low-rank matrix completion (RMC) has been studied extensively for computer vision, signal processing and machine learning applications. This problem aims to decompose a partially observed matrix into the superposition of a low-rank matrix and a sparse matrix, where the sparse matrix captures the grossly corrupted entries of the matrix. A widely used approach to tackle RMC is to consider a convex formulation, which minimizes the nuclear norm of the low-rank matrix (to promote low-rankness) and the l1 norm of the sparse matrix (to promote sparsity). In this paper, motivated by some recent works on low-
  arXiv  Detail & Related papers  (2020-08-18T04:46:22Z)
• Follow the bisector: a simple method for multi-objective optimization [65.83318707752385]
  We consider optimization problems, where multiple differentiable losses have to be minimized. The presented method computes descent direction in every iteration to guarantee equal relative decrease of objective functions.
  arXiv  Detail & Related papers  (2020-07-14T09:50:33Z)
• Optimal Structured Principal Subspace Estimation: Metric Entropy and Minimax Rates [6.00362077400694]
  This paper presents a unified framework for the statistical analysis of a general structured principal subspace estimation problem. It includes as special cases non-negative PCA/SVD, sparse PCA/SVD, subspace constrained PCA/SVD, and spectral clustering. Applying the general results to the specific settings yields the minimax rates of convergence for those problems.
  arXiv  Detail & Related papers  (2020-02-18T15:02:11Z)
• D-GCCA: Decomposition-based Generalized Canonical Correlation Analysis for Multi-view High-dimensional Data [11.184915338554422]
  A popular model in high-dimensional multi-view data analysis decomposes each view's data matrix into a low-rank common-source matrix generated by latent factors common across all data views. We propose a novel decomposition method for this model, called decomposition-based generalized canonical correlation analysis (D-GCCA) Our D-GCCA takes one step further than generalized canonical correlation analysis by separating common and distinctive components among canonical variables.
  arXiv  Detail & Related papers  (2020-01-09T06:35:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.