Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Compression of Deep Neural Networks

• URL: http://arxiv.org/abs/2606.00130v1
• Date: Thu, 28 May 2026 19:43:10 GMT
• Title: Automatically Differentiable Nonlinear Tensor Networks (ADNTNs) for Exponential Compression of Deep Neural Networks
• Abstract summary: ADNTNs are a family of structured weight generators with compact core tensors trained end-to-end by reverse-mode automatic differentiation (AD)<n>The paper focuses on three architectures: Tree Networks (TTNs), augmented TTNs (aTTNs) with boundary disentanglers, and Multi-scale Entanglement Renormalisation (MERA)
• Score: 8.605564917115663
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We study Automatically Differentiable Nonlinear Tensor Networks (ADNTNs), a family of structured weight generators whose compact core tensors are trained end-to-end by reverse-mode automatic differentiation (AD). The approach can be viewed as a natural extension of low-rank adaptation and tensor factorisation: instead of using one low-rank matrix update, an ADNTN builds a large weight tensor through a hierarchy of small cores, nonlinear activations, and optional lateral mixing tensors. The paper focuses on three architectures: Tree Tensor Networks (TTNs), augmented TTNs (aTTNs) with boundary disentanglers, and Multi-scale Entanglement Renormalisation Ansatze (MERA). The formulation supports nonlinear activations, task-aware objectives, batching, and hardware-aware execution schedules. At the same time, the paper keeps a clear distinction between \emph{differentiating} a contraction program and making contraction free: AD does not remove the cost of large intermediates, poor contraction orders, or exact contraction of general loopy tensor networks. Extensive simulations on AlexNet and VGG-16 layers show per-layer compression ratios from roughly $2000\times$ to $77000\times$ in the studied settings, with accuracy often matching the dense baseline and, in several VGG-16 cases, improving it. These results are encouraging rather than final: they suggest that ADNTNs are a promising, mathematically structured, and hardware-aware route toward much smaller neural networks, provided that optimisation, contraction schedules, and deployment kernels are designed together.

Related papers

• Renormalization Group Guided Tensor Network Structure Search [58.0378300612202]
  Network structure search (TN-SS) aims to automatically discover optimal network topologies and rank robustness for efficient tensor decomposition in high-dimensional data representation.<n>We propose RGTN (Renormalization Group guided Network search), a physics-inspired framework transforming TN-SS via multi-scale renormalization group flows.
  arXiv  Detail & Related papers  (2025-12-31T06:31:43Z)
• Tensor Decomposition Networks for Fast Machine Learning Interatomic Potential Computations [48.46721044282335]
  tensor decomposition networks (TDNs) achieve competitive performance with dramatic speedup in computations.<n>We evaluate TDNs on PubChemQCR, a newly curated molecular relaxation dataset containing 105 million DFT-calculated snapshots.<n>Results show that TDNs achieve competitive performance with dramatic speedup in computations.
  arXiv  Detail & Related papers  (2025-07-01T18:46:27Z)
• "Lossless" Compression of Deep Neural Networks: A High-dimensional Neural Tangent Kernel Approach [49.744093838327615]
  We provide a novel compression approach to wide and fully-connected emphdeep neural nets. Experiments on both synthetic and real-world data are conducted to support the advantages of the proposed compression scheme.
  arXiv  Detail & Related papers  (2024-03-01T03:46:28Z)
• Compressing Neural Networks Using Tensor Networks with Exponentially Fewer Variational Parameters [4.373746415510521]
  Neural network (NN) designed for challenging machine learning tasks is in general a highly nonlinear mapping that contains massive variational parameters.<n>We propose a general compression scheme that significantly reduces the variational parameters of NN's by encoding them to deep Rautomatically differentiable tensor network (ADTN)<n>Our work suggests deep TN as an exceptionally efficient mathematical structure for representing the variational parameters of NN's, which exhibits superior compressibility over the commonly-used matrices and multi-way arrays.
  arXiv  Detail & Related papers  (2023-05-10T11:24:27Z)
• Tensor Network States with Low-Rank Tensors [6.385624548310884]
  We introduce the idea of imposing low-rank constraints on the tensors that compose the tensor network. With this modification, the time and complexities for the network optimization can be substantially reduced. We find that choosing the tensor rank $r$ to be on the order of the bond $m$, is sufficient to obtain high-accuracy groundstate approximations.
  arXiv  Detail & Related papers  (2022-05-30T17:58:16Z)
• Nonlinear Tensor Ring Network [39.89070144585793]
  State-of-the-art deep neural networks (DNNs) have been widely applied for various real-world applications, and achieved significant performance for cognitive problems. By converting redundant models into compact ones, compression technique appears to be a practical solution to reducing the storage and memory consumption. In this paper, we develop a nonlinear tensor ring network (NTRN) in which both fullyconnected and convolutional layers are compressed.
  arXiv  Detail & Related papers  (2021-11-12T02:02:55Z)
• MLCTR: A Fast Scalable Coupled Tensor Completion Based on Multi-Layer Non-Linear Matrix Factorization [3.6978630614152013]
  This paper focuses on the embedding learning aspect of the tensor completion problem and proposes a new multi-layer neural network architecture for factorization and completion (MLCTR) The network architecture entails multiple advantages: a series of low-rank matrix factorizations building blocks to minimize overfitting, interleaved transfer functions in each layer for non-linearity, and by-pass connections to reduce diminishing problem and increase depths of networks. Our algorithm is highly efficient for imputing missing values in the EPS data.
  arXiv  Detail & Related papers  (2021-09-04T03:08:34Z)
• Learning N:M Fine-grained Structured Sparse Neural Networks From Scratch [75.69506249886622]
  Sparsity in Deep Neural Networks (DNNs) has been widely studied to compress and accelerate the models on resource-constrained environments. In this paper, we are the first to study training from scratch an N:M fine-grained structured sparse network.
  arXiv  Detail & Related papers  (2021-02-08T05:55:47Z)
• A Fully Tensorized Recurrent Neural Network [48.50376453324581]
  We introduce a "fully tensorized" RNN architecture which jointly encodes the separate weight matrices within each recurrent cell. This approach reduces model size by several orders of magnitude, while still maintaining similar or better performance compared to standard RNNs.
  arXiv  Detail & Related papers  (2020-10-08T18:24:12Z)
• Finite Versus Infinite Neural Networks: an Empirical Study [69.07049353209463]
  kernel methods outperform fully-connected finite-width networks. Centered and ensembled finite networks have reduced posterior variance. Weight decay and the use of a large learning rate break the correspondence between finite and infinite networks.
  arXiv  Detail & Related papers  (2020-07-31T01:57:47Z)
• DHP: Differentiable Meta Pruning via HyperNetworks [158.69345612783198]
  This paper introduces a differentiable pruning method via hypernetworks for automatic network pruning. Latent vectors control the output channels of the convolutional layers in the backbone network and act as a handle for the pruning of the layers. Experiments are conducted on various networks for image classification, single image super-resolution, and denoising.
  arXiv  Detail & Related papers  (2020-03-30T17:59:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.