Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering

• URL: http://arxiv.org/abs/2601.11626v1
• Date: Mon, 12 Jan 2026 18:15:53 GMT
• Title: Concatenated Matrix SVD: Compression Bounds, Incremental Approximation, and Error-Constrained Clustering
• Authors: Maksym Shamrai,
• Abstract summary: We propose three clustering algorithms that merge matrices only when their predicted joint SVD compression error remains below a user-specified threshold.<n>The algorithms span a trade-off between speed, provable accuracy, and scalability, enabling compression-aware clustering with explicit error control.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large collections of matrices arise throughout modern machine learning, signal processing, and scientific computing, where they are commonly compressed by concatenation followed by truncated singular value decomposition (SVD). This strategy enables parameter sharing and efficient reconstruction and has been widely adopted across domains ranging from multi-view learning and signal processing to neural network compression. However, it leaves a fundamental question unanswered: which matrices can be safely concatenated and compressed together under explicit reconstruction error constraints? Existing approaches rely on heuristic or architecture-specific grouping and provide no principled guarantees on the resulting SVD approximation error. In the present work, we introduce a theory-driven framework for compression-aware clustering of matrices under SVD compression constraints. Our analysis establishes new spectral bounds for horizontally concatenated matrices, deriving global upper bounds on the optimal rank-$r$ SVD reconstruction error from lower bounds on singular value growth. The first bound follows from Weyl-type monotonicity under blockwise extensions, while the second leverages singular values of incremental residuals to yield tighter, per-block guarantees. We further develop an efficient approximate estimator based on incremental truncated SVD that tracks dominant singular values without forming the full concatenated matrix. Therefore, we propose three clustering algorithms that merge matrices only when their predicted joint SVD compression error remains below a user-specified threshold. The algorithms span a trade-off between speed, provable accuracy, and scalability, enabling compression-aware clustering with explicit error control. Code is available online.

Related papers

• COMPOT: Calibration-Optimized Matrix Procrustes Orthogonalization for Transformers Compression [5.280540253822294]
  Post-training compression of Transformer models commonly relies on truncated singular value decomposition (SVD)<n>We propose COMPOT, a training-free compression framework that uses a small calibration dataset to estimate a sparse weight factorization.<n> COMPOT consistently delivers a superior quality-compression trade-off over strong low-rank and sparse baselines.
  arXiv  Detail & Related papers  (2026-02-16T21:31:34Z)
• Hierarchical Sparse Plus Low Rank Compression of LLM [2.4311207322523023]
  We present Hierarchical Sparse Plus Low-Rank (HSS) compression, a two-stage scheme that removes the largest-magnitude weights into a sparse matrix S.<n>HSS is hardware-friendly: its matrix-vector multiply reduces to one sparse and a sequence of thin-matrix multiplications.<n>Experiments on LLaMA-7B show that targeting only the self-attention projections suffices to yield large memory savings.
  arXiv  Detail & Related papers  (2025-12-19T04:28:30Z)
• NeuMatC: A General Neural Framework for Fast Parametric Matrix Operation [75.91285900600549]
  We propose textbftextitNeural Matrix Computation Framework (NeuMatC), which elegantly tackles general parametric matrix operation tasks.<n>NeuMatC unsupervisedly learns a low-rank and continuous mapping from parameters to their corresponding matrix operation results.<n> Experimental results on both synthetic and real-world datasets demonstrate the promising performance of NeuMatC.
  arXiv  Detail & Related papers  (2025-11-28T07:21:17Z)
• CPSVD: Enhancing Large Language Model Compression via Column-Preserving Singular Value Decomposition [20.302975265084964]
  We propose textbfColumn-textbfPreserving textbfSingular textbfValue textbfDecomposition (CPSVD)<n>CPSVD refines SVD-based Large Language Models compression by intelligently segmenting the parameter matrix.<n>It consistently outperforms state-of-the-art SVD-based LLM compression methods, achieving lower perplexity and higher accuracy on zero-shot tasks.
  arXiv  Detail & Related papers  (2025-10-22T09:02:37Z)
• COSPADI: Compressing LLMs via Calibration-Guided Sparse Dictionary Learning [5.595343998068235]
  CoSpaDi is a training-free compression framework that replaces low-rank decomposition with a more flexible structured sparse factorization.<n>We evaluate CoSpaDi across multiple Llama and Qwen models under per-layer and per-group settings at 20-50% compression ratios.
  arXiv  Detail & Related papers  (2025-09-26T08:55:09Z)
• Perturbation Analysis of Singular Values in Concatenated Matrices [0.0]
  How does the singular value spectrum and singular perturbationd matrix relate to the spectra of its individual components?<n>We setup analytical bounds that quantify stability of values under small perturbations in submatrices.<n>Results demonstrate that if submatrices are close in a norm, dominant singular values of the singular matrix remain stable enabling controlled trade-offs between accuracy and compression.
  arXiv  Detail & Related papers  (2025-03-11T09:28:57Z)
• DBA: Efficient Transformer with Dynamic Bilinear Low-Rank Attention [53.02648818164273]
  We present an efficient yet effective attention mechanism, namely the Dynamic Bilinear Low-Rank Attention (DBA) DBA compresses the sequence length by input-sensitive dynamic projection matrices and achieves linear time and space complexity. Experiments over tasks with diverse sequence length conditions show that DBA achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2022-11-24T03:06:36Z)
• Semi-Supervised Subspace Clustering via Tensor Low-Rank Representation [64.49871502193477]
  We propose a novel semi-supervised subspace clustering method, which is able to simultaneously augment the initial supervisory information and construct a discriminative affinity matrix. Comprehensive experimental results on six commonly-used benchmark datasets demonstrate the superiority of our method over state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-05-21T01:47:17Z)
• Solving weakly supervised regression problem using low-rank manifold regularization [77.34726150561087]
  We solve a weakly supervised regression problem. Under "weakly" we understand that for some training points the labels are known, for some unknown, and for others uncertain due to the presence of random noise or other reasons such as lack of resources. In the numerical section, we applied the suggested method to artificial and real datasets using Monte-Carlo modeling.
  arXiv  Detail & Related papers  (2021-04-13T23:21:01Z)
• A Scalable, Adaptive and Sound Nonconvex Regularizer for Low-rank Matrix Completion [60.52730146391456]
  We propose a new non scalable low-rank regularizer called "nuclear Frobenius norm" regularizer, which is adaptive and sound. It bypasses the computation of singular values and allows fast optimization by algorithms. It obtains state-of-the-art recovery performance while being the fastest in existing matrix learning methods.
  arXiv  Detail & Related papers  (2020-08-14T18:47:58Z)
• Multi-Objective Matrix Normalization for Fine-grained Visual Recognition [153.49014114484424]
  Bilinear pooling achieves great success in fine-grained visual recognition (FGVC) Recent methods have shown that the matrix power normalization can stabilize the second-order information in bilinear features. We propose an efficient Multi-Objective Matrix Normalization (MOMN) method that can simultaneously normalize a bilinear representation.
  arXiv  Detail & Related papers  (2020-03-30T08:40:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.