Multiscale Tensor Summation Factorization as a New Neural Network Layer (MTS Layer) for Multidimensional Data Processing

• URL: http://arxiv.org/abs/2504.13975v1
• Date: Thu, 17 Apr 2025 22:19:59 GMT
• Title: Multiscale Tensor Summation Factorization as a New Neural Network Layer (MTS Layer) for Multidimensional Data Processing
• Authors: Mehmet Yamaç, Muhammad Numan Yousaf, Serkan Kiranyaz, Moncef Gabbouj,
• Abstract summary: Multiscale Summation (MTS) Factorization is a novel neural network operator that implements tensor summation at multiple scales.<n> MTS not only reduces the number of parameters required while enhancing the efficiency of weight optimization, but it also demonstrates clear advantages over convolutional layers.<n>The corresponding neural network, MTSNet, demonstrates a more favorable complexity-performance tradeoff compared to state-of-the-art transformers in various computer vision applications.
• Score: 18.557169937152967
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Multilayer perceptrons (MLP), or fully connected artificial neural networks, are known for performing vector-matrix multiplications using learnable weight matrices; however, their practical application in many machine learning tasks, especially in computer vision, can be limited due to the high dimensionality of input-output pairs at each layer. To improve efficiency, convolutional operators have been utilized to facilitate weight sharing and local connections, yet they are constrained by limited receptive fields. In this paper, we introduce Multiscale Tensor Summation (MTS) Factorization, a novel neural network operator that implements tensor summation at multiple scales, where each tensor to be summed is obtained through Tucker-decomposition-like mode products. Unlike other tensor decomposition methods in the literature, MTS is not introduced as a network compression tool; instead, as a new backbone neural layer. MTS not only reduces the number of parameters required while enhancing the efficiency of weight optimization compared to traditional dense layers (i.e., unfactorized weight matrices in MLP layers), but it also demonstrates clear advantages over convolutional layers. The proof-of-concept experimental comparison of the proposed MTS networks with MLPs and Convolutional Neural Networks (CNNs) demonstrates their effectiveness across various tasks, such as classification, compression, and signal restoration. Additionally, when integrated with modern non-linear units such as the multi-head gate (MHG), also introduced in this study, the corresponding neural network, MTSNet, demonstrates a more favorable complexity-performance tradeoff compared to state-of-the-art transformers in various computer vision applications. The software implementation of the MTS layer and the corresponding MTS-based networks, MTSNets, is shared at https://github.com/mehmetyamac/MTSNet.

Related papers

• Deep-Unrolling Multidimensional Harmonic Retrieval Algorithms on Neuromorphic Hardware [78.17783007774295]
  This paper explores the potential of conversion-based neuromorphic algorithms for highly accurate and energy-efficient single-snapshot multidimensional harmonic retrieval. A novel method for converting the complex-valued convolutional layers and activations into spiking neural networks (SNNs) is developed. The converted SNNs achieve almost five-fold power efficiency at moderate performance loss compared to the original CNNs.
  arXiv  Detail & Related papers  (2024-12-05T09:41:33Z)
• Structured and Balanced Multi-component and Multi-layer Neural Networks [9.699640804685629]
  We propose a balanced multi-component and multi-layer computation network (MMNN) MMNNs achieve a significant reduction of training parameters compared to fully connected neural networks (FCNNs) or multi-layer perceptrons (MLPs)
  arXiv  Detail & Related papers  (2024-06-30T17:00:42Z)
• A Matrix Product State Model for Simultaneous Classification and Generation [0.8192907805418583]
  Quantum machine learning (QML) is a rapidly expanding field that merges the principles of quantum computing with the techniques of machine learning. Here, we present a novel matrix product state (MPS) model, where the MPS functions as both a classifier and a generator. Our contributions offer insights into the mechanics of tensor network methods for generation tasks.
  arXiv  Detail & Related papers  (2024-06-25T10:23:36Z)
• conv_einsum: A Framework for Representation and Fast Evaluation of Multilinear Operations in Convolutional Tensorial Neural Networks [28.416123889998243]
  We develop a framework for representing tensorial convolution layers as einsum-like strings and a meta-algorithm conv_einsum which is able to evaluate these strings in a FLOPs-minimizing manner.
  arXiv  Detail & Related papers  (2024-01-07T04:30:12Z)
• Deformable Mixer Transformer with Gating for Multi-Task Learning of Dense Prediction [126.34551436845133]
  CNNs and Transformers have their own advantages and both have been widely used for dense prediction in multi-task learning (MTL) We present a novel MTL model by combining both merits of deformable CNN and query-based Transformer with shared gating for multi-task learning of dense prediction.
  arXiv  Detail & Related papers  (2023-08-10T17:37:49Z)
• BiTAT: Neural Network Binarization with Task-dependent Aggregated Transformation [116.26521375592759]
  Quantization aims to transform high-precision weights and activations of a given neural network into low-precision weights/activations for reduced memory usage and computation. Extreme quantization (1-bit weight/1-bit activations) of compactly-designed backbone architectures results in severe performance degeneration. This paper proposes a novel Quantization-Aware Training (QAT) method that can effectively alleviate performance degeneration.
  arXiv  Detail & Related papers  (2022-07-04T13:25:49Z)
• Low-bit Quantization of Recurrent Neural Network Language Models Using Alternating Direction Methods of Multipliers [67.688697838109]
  This paper presents a novel method to train quantized RNNLMs from scratch using alternating direction methods of multipliers (ADMM) Experiments on two tasks suggest the proposed ADMM quantization achieved a model size compression factor of up to 31 times over the full precision baseline RNNLMs.
  arXiv  Detail & Related papers  (2021-11-29T09:30:06Z)
• Compact representations of convolutional neural networks via weight pruning and quantization [63.417651529192014]
  We propose a novel storage format for convolutional neural networks (CNNs) based on source coding and leveraging both weight pruning and quantization. We achieve a reduction of space occupancy up to 0.6% on fully connected layers and 5.44% on the whole network, while performing at least as competitive as the baseline.
  arXiv  Detail & Related papers  (2021-08-28T20:39:54Z)
• Residual Matrix Product State for Machine Learning [20.158215120846652]
  We propose the residual matrix product state (ResMPS) by combining the ideas of matrix product state (MPS) and residual neural network (NN) ResMPS can be treated as a network where its layers map the "hidden" features to the outputs. It outperforms state-of-the-art TN models on efficiency, stability and expression power.
  arXiv  Detail & Related papers  (2020-12-22T05:44:20Z)
• Block-term Tensor Neural Networks [29.442026567710435]
  We show that block-term tensor layers (BT-layers) can be easily adapted to neural network models, such as CNNs and RNNs. BT-layers in CNNs and RNNs can achieve a very large compression ratio on the number of parameters while preserving or improving the representation power of the original DNNs.
  arXiv  Detail & Related papers  (2020-10-10T09:58:43Z)
• Tensor train decompositions on recurrent networks [60.334946204107446]
  Matrix product state (MPS) tensor trains have more attractive features than MPOs, in terms of storage reduction and computing time at inference. We show that MPS tensor trains should be at the forefront of LSTM network compression through a theoretical analysis and practical experiments on NLP task.
  arXiv  Detail & Related papers  (2020-06-09T18:25:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.