Robust Basis Spline Decoupling for the Compression of Transformer Models

• URL: http://arxiv.org/abs/2605.18794v1
• Date: Mon, 11 May 2026 06:37:36 GMT
• Title: Robust Basis Spline Decoupling for the Compression of Transformer Models
• Abstract summary: We introduce a B-spline-based decoupling framework that generalizes existing approaches.<n>By exploiting the local support and flexible smoothness control of B-splines, the proposed formulation yields a more stable and expressive representation.
• Score: 1.1470070927586018
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Decoupling is a powerful modeling paradigm for representing multivariate functions as compositions of linear transformations and univariate nonlinear functions. A single-layer decoupling can be viewed as a fully connected neural network with a single hidden layer and flexible activation functions, providing a direct link with neural networks. Because of this, the use of decoupling methods has gained increasing attention in neural network domains, particularly compression, since it enables structured approximations with reduced parameter complexity. Existing tensor-based decoupling methods typically rely on polynomial or piecewise-linear parameterizations of the internal nonlinear functions, which can suffer from numerical instability or limited expressiveness. In this work, we introduce a B-spline-based decoupling framework that generalizes these existing approaches. By exploiting the local support and flexible smoothness control of B-splines, the proposed formulation yields a more numerically stable and expressive representation. We derive a constrained coupled matrix-tensor factorization and propose a robust alternating least-squares algorithm, called R-CMTF-BSD, incorporating normalization and Tikhonov regularization. The proposed method is validated through experiments on synthetic data and transformer model compression. Results on the Vision and Swin Transformer architectures demonstrate that B-spline decoupling enables substantial parameter reduction while maintaining competitive accuracy, making the R-CMTF-BSD algorithm a promising tool for structured neural network compression.

Related papers

• Trainable Spline Representations for Physics-Informed Learning [0.0]
  Physics-Informed Splines (PI-Splines) is a structured spline-based architecture for physics-informed learning.<n>PI-Splines provide a competitive and stable alternative to neural physics-informed architectures.
  arXiv  Detail & Related papers  (2026-07-17T08:39:31Z)
• Neural Network Compression by Approximate Differential Equivalence [0.9099663022952496]
  We propose an approach that compresses models by aggregating neurons rather than removing weights independently.<n>Our method encodes a trained network as induces a smooth ODE system and applies a lumping method called Approximate Forward Differential Equivalence.<n>We evaluate the method on synthetic datasets derived from nonlinear systems with known ground-truth behavior and on public regression benchmarks.
  arXiv  Detail & Related papers  (2026-05-31T19:01:46Z)
• Multi-Scale Separable Fourier Neural Networks for Solving High-Frequency PDEs [2.7347781746626705]
  We propose a novel neural network architecture, Multi-Scale Separable Neural Networks (MS-SFNN)<n>MS-SFNN achieves unprecedented accuracy across a range of challenging PDEs, significantly outperforming state-of-the-art methods.
  arXiv  Detail & Related papers  (2026-05-29T08:58:39Z)
• DualFlexKAN: Dual-stage Kolmogorov-Arnold Networks with Independent Function Control [2.071516130824992]
  Kolmogorov-Arnold Networks (KANs) address this limitation through edge-centric learnable functions.<n>KANs suffer from parameter scaling and architectural rigidity that hinders the effective integration of standard regularization techniques.<n>This paper introduces the DualFlexKAN (DFKAN), a flexible architecture featuring a dual-stage mechanism that independently controls pre-linear input transformations and post-linear outputs.
  arXiv  Detail & Related papers  (2026-03-09T16:36:04Z)
• Low-Rank Tensor Recovery via Variational Schatten-p Quasi-Norm and Jacobian Regularization [49.85875869048434]
  We propose a CP-based low-rank tensor function parameterized by neural networks for implicit neural representation.<n>To achieve sparser CP decomposition, we introduce a variational Schatten-p quasi-norm to prune redundant rank-1 components.<n>For smoothness, we propose a regularization term based on the spectral norm of the Jacobian and Hutchinson's trace estimator.
  arXiv  Detail & Related papers  (2025-06-27T11:23:10Z)
• Adaptive Error-Bounded Hierarchical Matrices for Efficient Neural Network Compression [0.0]
  This paper introduces a dynamic, error-bounded hierarchical matrix (H-matrix) compression method tailored for Physics-Informed Neural Networks (PINNs) The proposed approach reduces the computational complexity and memory demands of large-scale physics-based models while preserving the essential properties of the Neural Tangent Kernel (NTK) Empirical results demonstrate that this technique outperforms traditional compression methods, such as Singular Value Decomposition (SVD), pruning, and quantization, by maintaining high accuracy and improving generalization capabilities.
  arXiv  Detail & Related papers  (2024-09-11T05:55:51Z)
• The Convex Landscape of Neural Networks: Characterizing Global Optima and Stationary Points via Lasso Models [75.33431791218302]
  Deep Neural Network Network (DNN) models are used for programming purposes. In this paper we examine the use of convex neural recovery models. We show that all the stationary non-dimensional objective objective can be characterized as the standard a global subsampled convex solvers program. We also show that all the stationary non-dimensional objective objective can be characterized as the standard a global subsampled convex solvers program.
  arXiv  Detail & Related papers  (2023-12-19T23:04:56Z)
• Scalable Neural Network Kernels [22.299704296356836]
  We introduce scalable neural network kernels (SNNKs), capable of approximating regular feedforward layers (FFLs) We also introduce the neural network bundling process that applies SNNKs to compactify deep neural network architectures. Our mechanism provides up to 5x reduction in the number of trainable parameters, while maintaining competitive accuracy.
  arXiv  Detail & Related papers  (2023-10-20T02:12:56Z)
• Decoupling multivariate functions using a nonparametric filtered tensor decomposition [0.29360071145551075]
  Decoupling techniques aim at providing an alternative representation of the nonlinearity. The so-called decoupled form is often a more efficient parameterisation of the relationship while being highly structured, favouring interpretability. In this work two new algorithms, based on filtered tensor decompositions of first order derivative information are introduced.
  arXiv  Detail & Related papers  (2022-05-23T09:34:17Z)
• CSformer: Bridging Convolution and Transformer for Compressive Sensing [65.22377493627687]
  This paper proposes a hybrid framework that integrates the advantages of leveraging detailed spatial information from CNN and the global context provided by transformer for enhanced representation learning. The proposed approach is an end-to-end compressive image sensing method, composed of adaptive sampling and recovery. The experimental results demonstrate the effectiveness of the dedicated transformer-based architecture for compressive sensing.
  arXiv  Detail & Related papers  (2021-12-31T04:37:11Z)
• Rate Distortion Characteristic Modeling for Neural Image Compression [59.25700168404325]
  End-to-end optimization capability offers neural image compression (NIC) superior lossy compression performance. distinct models are required to be trained to reach different points in the rate-distortion (R-D) space. We make efforts to formulate the essential mathematical functions to describe the R-D behavior of NIC using deep network and statistical modeling.
  arXiv  Detail & Related papers  (2021-06-24T12:23:05Z)
• Provably Efficient Neural Estimation of Structural Equation Model: An Adversarial Approach [144.21892195917758]
  We study estimation in a class of generalized Structural equation models (SEMs) We formulate the linear operator equation as a min-max game, where both players are parameterized by neural networks (NNs), and learn the parameters of these neural networks using a gradient descent. For the first time we provide a tractable estimation procedure for SEMs based on NNs with provable convergence and without the need for sample splitting.
  arXiv  Detail & Related papers  (2020-07-02T17:55:47Z)
• Supervised Learning for Non-Sequential Data: A Canonical Polyadic Decomposition Approach [85.12934750565971]
  Efficient modelling of feature interactions underpins supervised learning for non-sequential tasks. To alleviate this issue, it has been proposed to implicitly represent the model parameters as a tensor. For enhanced expressiveness, we generalize the framework to allow feature mapping to arbitrarily high-dimensional feature vectors.
  arXiv  Detail & Related papers  (2020-01-27T22:38:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.