Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory

• URL: http://arxiv.org/abs/2512.12911v1
• Date: Mon, 15 Dec 2025 01:49:20 GMT
• Title: Evaluating Singular Value Thresholds for DNN Weight Matrices based on Random Matrix Theory
• Authors: Kohei Nishikawa, Koki Shimizu, Hashiguchi Hiroki,
• Abstract summary: We evaluate thresholds for removing singular values from singular value decomposition-based low-rank approximations of deep neural network weight matrices.<n>The proposed metric is used in numerical experiments to compare two threshold estimation methods.
• Score: 0.0
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This study evaluates thresholds for removing singular values from singular value decomposition-based low-rank approximations of deep neural network weight matrices. Each weight matrix is modeled as the sum of signal and noise matrices. The low-rank approximation is obtained by removing noise-related singular values using a threshold based on random matrix theory. To assess the adequacy of this threshold, we propose an evaluation metric based on the cosine similarity between the singular vectors of the signal and original weight matrices. The proposed metric is used in numerical experiments to compare two threshold estimation methods.

Related papers

• Near-optimal Rank Adaptive Inference of High Dimensional Matrices [46.66027208538566]
  We address the problem of estimating a high-dimensional matrix from linear measurements.<n>We propose an algorithm that combines a Least-Squares estimator with a universal singular value thresholding procedure.<n>Our results rely on an enhanced analysis of matrix denoising methods based on singular value thresholding.
  arXiv  Detail & Related papers  (2025-10-09T12:01:46Z)
• Non-Asymptotic Analysis of Data Augmentation for Precision Matrix Estimation [12.919305286055616]
  We focus on two classes of estimators: linear shrinkage estimators with a target proportional to the identity matrix, and estimators derived from data augmentation.<n>For both classes of estimators, we derive estimators and provide concentration bounds for their quadratic error.<n>On the technical side, our analysis relies on tools from random matrix theory.
  arXiv  Detail & Related papers  (2025-10-02T15:28:14Z)
• Extreme value theory for singular subspace estimation in the matrix denoising model [0.4297070083645049]
  We study fine-grained singular subspace estimation in the matrix denoising model.<n>We apply our distributional theory to test hypotheses of low-rank signal structure encoded in the leading singular vectors.
  arXiv  Detail & Related papers  (2025-07-26T15:28:36Z)
• A simple estimator of the correlation kernel matrix of a determinantal point process [3.692410936160711]
  This paper proposes a closed form estimator of the Determinantal Point Process (DPP)<n>We prove the consistency and normality of our estimator, as well as its large deviation properties.
  arXiv  Detail & Related papers  (2025-05-20T15:48:45Z)
• Entrywise error bounds for low-rank approximations of kernel matrices [55.524284152242096]
  We derive entrywise error bounds for low-rank approximations of kernel matrices obtained using the truncated eigen-decomposition. A key technical innovation is a delocalisation result for the eigenvectors of the kernel matrix corresponding to small eigenvalues. We validate our theory with an empirical study of a collection of synthetic and real-world datasets.
  arXiv  Detail & Related papers  (2024-05-23T12:26:25Z)
• Spectral Entry-wise Matrix Estimation for Low-Rank Reinforcement Learning [53.445068584013896]
  We study matrix estimation problems arising in reinforcement learning (RL) with low-rank structure. In low-rank bandits, the matrix to be recovered specifies the expected arm rewards, and for low-rank Markov Decision Processes (MDPs), it may for example characterize the transition kernel of the MDP. We show that simple spectral-based matrix estimation approaches efficiently recover the singular subspaces of the matrix and exhibit nearly-minimal entry-wise error.
  arXiv  Detail & Related papers  (2023-10-10T17:06:41Z)
• An approach to robust ICP initialization [77.45039118761837]
  We propose an approach to initialize the Iterative Closest Point (ICP) algorithm to match unlabelled point clouds related by rigid transformations. We derive bounds on the robustness of our approach to noise and numerical experiments confirm our theoretical findings.
  arXiv  Detail & Related papers  (2022-12-10T16:27:25Z)
• Kernel Density Estimation by Stagewise Algorithm with a Simple Dictionary [0.0]
  This paper studies kernel density estimation by stagewise algorithm with a simple dictionary on U-divergence. We randomly split an i.i.d. sample into two disjoint sets, one for constructing the kernels in the dictionary and the other for evaluating the estimator.
  arXiv  Detail & Related papers  (2021-07-27T17:05:06Z)
• Adversarially-Trained Nonnegative Matrix Factorization [77.34726150561087]
  We consider an adversarially-trained version of the nonnegative matrix factorization. In our formulation, an attacker adds an arbitrary matrix of bounded norm to the given data matrix. We design efficient algorithms inspired by adversarial training to optimize for dictionary and coefficient matrices.
  arXiv  Detail & Related papers  (2021-04-10T13:13:17Z)
• Learning Noise Transition Matrix from Only Noisy Labels via Total Variation Regularization [88.91872713134342]
  We propose a theoretically grounded method that can estimate the noise transition matrix and learn a classifier simultaneously. We show the effectiveness of the proposed method through experiments on benchmark and real-world datasets.
  arXiv  Detail & Related papers  (2021-02-04T05:09:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.