Ridges, Neural Networks, and the Radon Transform

• URL: http://arxiv.org/abs/2203.02543v1
• Date: Fri, 4 Mar 2022 19:38:15 GMT
• Title: Ridges, Neural Networks, and the Radon Transform
• Authors: Michael Unser
• Abstract summary: Ridges appear in the theory of neural networks as functional descriptors of the effect of a neuron. We investigate properties of the Radon transform in relation to ridges and to the characterization of neural networks.
• Score: 25.6264886382888
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A ridge is a function that is characterized by a one-dimensional profile (activation) and a multidimensional direction vector. Ridges appear in the theory of neural networks as functional descriptors of the effect of a neuron, with the direction vector being encoded in the linear weights. In this paper, we investigate properties of the Radon transform in relation to ridges and to the characterization of neural networks. We introduce a broad category of hyper-spherical Banach subspaces (including the relevant subspace of measures) over which the back-projection operator is invertible. We also give conditions under which the back-projection operator is extendable to the full parent space with its null space being identifiable as a Banach complement. Starting from first principles, we then characterize the sampling functionals that are in the range of the filtered Radon transform. Next, we extend the definition of ridges for any distributional profile and determine their (filtered) Radon transform in full generality. Finally, we apply our formalism to clarify and simplify some of the results and proofs on the optimality of ReLU networks that have appeared in the literature.

Related papers

• Emergence of Grid-like Representations by Training Recurrent Networks with Conformal Normalization [48.99772993899573]
  We study the emergence of hexagon grid patterns of grid cells based on a general recurrent neural network model. We propose a simple yet general conformal normalization of the input velocity of the RNN. We conduct extensive experiments to verify that conformal normalization is crucial for the emergence of hexagon grid patterns.
  arXiv  Detail & Related papers  (2023-10-29T23:12:56Z)
• Joint Group Invariant Functions on Data-Parameter Domain Induce Universal Neural Networks [14.45619075342763]
  We present a systematic method to induce a generalized neural network and its right inverse operator, called the ridgelet transform. Since the ridgelet transform is an inverse, it can describe the arrangement of parameters for the network to represent a target function. We present a new simple proof of the universality by using Schur's lemma in a unified manner covering a wide class of networks.
  arXiv  Detail & Related papers  (2023-10-05T13:30:37Z)
• NeuRBF: A Neural Fields Representation with Adaptive Radial Basis Functions [93.02515761070201]
  We present a novel type of neural fields that uses general radial bases for signal representation. Our method builds upon general radial bases with flexible kernel position and shape, which have higher spatial adaptivity and can more closely fit target signals. When applied to neural radiance field reconstruction, our method achieves state-of-the-art rendering quality, with small model size and comparable training speed.
  arXiv  Detail & Related papers  (2023-09-27T06:32:05Z)
• Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networks [0.0]
  We prove a conjecture of Ongie et al. that every continuous piecewise linear function expressible with this kind of infinite width neural network is expressible as a finite width shallow ReLU neural network.
  arXiv  Detail & Related papers  (2023-07-25T15:38:18Z)
• On the Effective Number of Linear Regions in Shallow Univariate ReLU Networks: Convergence Guarantees and Implicit Bias [50.84569563188485]
  We show that gradient flow converges in direction when labels are determined by the sign of a target network with $r$ neurons. Our result may already hold for mild over- parameterization, where the width is $tildemathcalO(r)$ and independent of the sample size.
  arXiv  Detail & Related papers  (2022-05-18T16:57:10Z)
• Exploring Linear Feature Disentanglement For Neural Networks [63.20827189693117]
  Non-linear activation functions, e.g., Sigmoid, ReLU, and Tanh, have achieved great success in neural networks (NNs) Due to the complex non-linear characteristic of samples, the objective of those activation functions is to project samples from their original feature space to a linear separable feature space. This phenomenon ignites our interest in exploring whether all features need to be transformed by all non-linear functions in current typical NNs.
  arXiv  Detail & Related papers  (2022-03-22T13:09:17Z)
• Understanding neural networks with reproducing kernel Banach spaces [20.28372804772848]
  Characterizing function spaces corresponding to neural networks can provide a way to understand their properties. We prove a representer theorem for a wide class of reproducing kernel Banach spaces. For a suitable class of ReLU activation functions, the norm in the corresponding kernel Banach space can be characterized in terms of the inverse Radon transform of a bounded real measure.
  arXiv  Detail & Related papers  (2021-09-20T17:32:30Z)
• Convolutional Filtering and Neural Networks with Non Commutative Algebras [153.20329791008095]
  We study the generalization of non commutative convolutional neural networks. We show that non commutative convolutional architectures can be stable to deformations on the space of operators.
  arXiv  Detail & Related papers  (2021-08-23T04:22:58Z)
• Two-layer neural networks with values in a Banach space [1.90365714903665]
  We study two-layer neural networks whose domain and range are Banach spaces with separable preduals. As the nonlinearity we choose the lattice operation of taking the positive part; in case of $mathbb Rd$-valued neural networks this corresponds to the ReLU activation function.
  arXiv  Detail & Related papers  (2021-05-05T14:54:24Z)
• Approximations with deep neural networks in Sobolev time-space [5.863264019032882]
  Solution of evolution equation generally lies in certain Bochner-Sobolev spaces. Deep neural networks can approximate Sobolev-regular functions with respect to Bochner-Sobolev spaces.
  arXiv  Detail & Related papers  (2020-12-23T22:21:05Z)
• From deep to Shallow: Equivalent Forms of Deep Networks in Reproducing Kernel Krein Space and Indefinite Support Vector Machines [63.011641517977644]
  We take a deep network and convert it to an equivalent (indefinite) kernel machine. We then investigate the implications of this transformation for capacity control and uniform convergence. Finally, we analyse the sparsity properties of the flat representation, showing that the flat weights are (effectively) Lp-"norm" regularised with 0p1.
  arXiv  Detail & Related papers  (2020-07-15T03:21:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.