High-Order Tensor Regression in Sparse Convolutional Neural Networks

• URL: http://arxiv.org/abs/2501.01239v3
• Date: Mon, 06 Jan 2025 01:56:56 GMT
• Title: High-Order Tensor Regression in Sparse Convolutional Neural Networks
• Authors: Roberto Dias Algarte,
• Abstract summary: This article presents a generic approach to convolution that significantly differs from conventional methodologies in the current Machine Learning literature. A rational theory of regression in neural networks is developed, as a framework for a generic view of sparse convolutional neural networks. As a direct outcome, the classic Backpropagation Algorithm is redefined to align with this rational tensor-based approach and presented in its simplest, most generic form.
• Score: 0.0
• License:
• Abstract: This article presents a generic approach to convolution that significantly differs from conventional methodologies in the current Machine Learning literature. The approach, in its mathematical aspects, proved to be clear and concise, particularly when high-order tensors are involved. In this context, a rational theory of regression in neural networks is developed, as a framework for a generic view of sparse convolutional neural networks, the primary focus of this study. As a direct outcome, the classic Backpropagation Algorithm is redefined to align with this rational tensor-based approach and presented in its simplest, most generic form.

Related papers

• Tensor-Based Foundations of Ordinary Least Squares and Neural Network Regression Models [0.0]
  This article introduces a novel approach to the mathematical development of Ordinary Least Squares and Neural Network regression models. By leveraging Analysis and fundamental matrix computations, the theoretical foundations of both models are meticulously detailed and extended to their complete algorithmic forms.
  arXiv  Detail & Related papers  (2024-11-19T21:36:04Z)
• Understanding Deep Learning via Notions of Rank [5.439020425819001]
  This thesis puts forth notions of rank as key for developing a theory of deep learning. In particular, we establish that gradient-based training can induce an implicit regularization towards low rank for several neural network architectures. Practical implications of our theory for designing explicit regularization schemes and data preprocessing algorithms are presented.
  arXiv  Detail & Related papers  (2024-08-04T18:47:55Z)
• A singular Riemannian Geometry Approach to Deep Neural Networks III. Piecewise Differentiable Layers and Random Walks on $n$-dimensional Classes [49.32130498861987]
  We study the case of non-differentiable activation functions, such as ReLU. Two recent works introduced a geometric framework to study neural networks. We illustrate our findings with some numerical experiments on classification of images and thermodynamic problems.
  arXiv  Detail & Related papers  (2024-04-09T08:11:46Z)
• Graph Neural Networks for Learning Equivariant Representations of Neural Networks [55.04145324152541]
  We propose to represent neural networks as computational graphs of parameters. Our approach enables a single model to encode neural computational graphs with diverse architectures. We showcase the effectiveness of our method on a wide range of tasks, including classification and editing of implicit neural representations.
  arXiv  Detail & Related papers  (2024-03-18T18:01:01Z)
• When Deep Learning Meets Polyhedral Theory: A Survey [6.899761345257773]
  In the past decade, deep became the prevalent methodology for predictive modeling thanks to the remarkable accuracy of deep neural learning. Meanwhile, the structure of neural networks converged back to simplerwise and linear functions.
  arXiv  Detail & Related papers  (2023-04-29T11:46:53Z)
• Data-driven emergence of convolutional structure in neural networks [83.4920717252233]
  We show how fully-connected neural networks solving a discrimination task can learn a convolutional structure directly from their inputs. By carefully designing data models, we show that the emergence of this pattern is triggered by the non-Gaussian, higher-order local structure of the inputs.
  arXiv  Detail & Related papers  (2022-02-01T17:11:13Z)
• Implicit Regularization in Hierarchical Tensor Factorization and Deep Convolutional Neural Networks [18.377136391055327]
  This paper theoretically analyzes the implicit regularization in hierarchical tensor factorization. It translates to an implicit regularization towards locality for the associated convolutional networks. Our work highlights the potential of enhancing neural networks via theoretical analysis of their implicit regularization.
  arXiv  Detail & Related papers  (2022-01-27T18:48:30Z)
• Compressive Sensing and Neural Networks from a Statistical Learning Perspective [4.561032960211816]
  We present a generalization error analysis for a class of neural networks suitable for sparse reconstruction from few linear measurements. Under realistic conditions, the generalization error scales only logarithmically in the number of layers, and at most linear in number of measurements.
  arXiv  Detail & Related papers  (2020-10-29T15:05:43Z)
• Complexity-based speciation and genotype representation for neuroevolution [81.21462458089142]
  This paper introduces a speciation principle for neuroevolution where evolving networks are grouped into species based on the number of hidden neurons. The proposed speciation principle is employed in several techniques designed to promote and preserve diversity within species and in the ecosystem as a whole.
  arXiv  Detail & Related papers  (2020-10-11T06:26:56Z)
• Developing Constrained Neural Units Over Time [81.19349325749037]
  This paper focuses on an alternative way of defining Neural Networks, that is different from the majority of existing approaches. The structure of the neural architecture is defined by means of a special class of constraints that are extended also to the interaction with data. The proposed theory is cast into the time domain, in which data are presented to the network in an ordered manner.
  arXiv  Detail & Related papers  (2020-09-01T09:07:25Z)
• Understanding Generalization in Deep Learning via Tensor Methods [53.808840694241]
  We advance the understanding of the relations between the network's architecture and its generalizability from the compression perspective. We propose a series of intuitive, data-dependent and easily-measurable properties that tightly characterize the compressibility and generalizability of neural networks.
  arXiv  Detail & Related papers  (2020-01-14T22:26:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.