Integral representations of shallow neural network with Rectified Power Unit activation function

• URL: http://arxiv.org/abs/2112.11157v1
• Date: Mon, 20 Dec 2021 15:18:11 GMT
• Title: Integral representations of shallow neural network with Rectified Power Unit activation function
• Authors: Ahmed Abdeljawad, Philipp Grohs
• Abstract summary: We derive a formula for the integral representation of a shallow neural network with the Rectified Power Unit activation function. The multidimensional result in this paper characterizes the set of functions that can be represented with bounded norm and possibly unbounded width.
• Score: 5.863264019032882
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this effort, we derive a formula for the integral representation of a shallow neural network with the Rectified Power Unit activation function. Mainly, our first result deals with the univariate case of representation capability of RePU shallow networks. The multidimensional result in this paper characterizes the set of functions that can be represented with bounded norm and possibly unbounded width.

Related papers

• Activation Functions for "A Feedforward Unitary Equivariant Neural Network" [0.0]
  In previous work, we presented a feedforward unitary equivariant neural network. We proposed three distinct activation functions tailored for this network. This short paper generalises these activation functions to a single functional form.
  arXiv  Detail & Related papers  (2024-11-17T09:46:52Z)
• Separation Power of Equivariant Neural Networks [11.906285279109477]
  We propose a theoretical framework to investigate the separation power of equivariant neural networks with point-wise activations. We show that all non-polynomial activations, such as ReLU and sigmoid, are equivalent in terms of expressivity.
  arXiv  Detail & Related papers  (2024-06-13T09:52:44Z)
• Hamiltonian Mechanics of Feature Learning: Bottleneck Structure in Leaky ResNets [58.460298576330835]
  We study Leaky ResNets, which interpolate between ResNets and Fully-Connected nets depending on an 'effective depth' We leverage this intuition to explain the emergence of a bottleneck structure, as observed in previous work.
  arXiv  Detail & Related papers  (2024-05-27T18:15:05Z)
• Image segmentation with traveling waves in an exactly solvable recurrent neural network [71.74150501418039]
  We show that a recurrent neural network can effectively divide an image into groups according to a scene's structural characteristics. We present a precise description of the mechanism underlying object segmentation in this network. We then demonstrate a simple algorithm for object segmentation that generalizes across inputs ranging from simple geometric objects in grayscale images to natural images.
  arXiv  Detail & Related papers  (2023-11-28T16:46:44Z)
• Extending Neural Network Verification to a Larger Family of Piece-wise Linear Activation Functions [0.0]
  We extend an available neural network verification technique to support a wider class of piece-wise linear activation functions. We also extend the algorithms, which provide in their original form exact respectively over-approximative results for bounded input sets represented as start sets, to allow also unbounded input set.
  arXiv  Detail & Related papers  (2023-11-16T11:01:39Z)
• Generalized Activation via Multivariate Projection [46.837481855573145]
  Activation functions are essential to introduce nonlinearity into neural networks. We consider ReLU as a projection from R onto the nonnegative half-line R+. We extend ReLU by substituting it with a generalized projection operator onto a convex cone, such as the Second-Order Cone (SOC) projection.
  arXiv  Detail & Related papers  (2023-09-29T12:44:27Z)
• Piecewise Linear Functions Representable with Infinite Width Shallow ReLU Neural Networks [0.0]
  We prove a conjecture of Ongie et al. that every continuous piecewise linear function expressible with this kind of infinite width neural network is expressible as a finite width shallow ReLU neural network.
  arXiv  Detail & Related papers  (2023-07-25T15:38:18Z)
• Unification of popular artificial neural network activation functions [0.0]
  We present a unified representation of the most popular neural network activation functions. Adopting Mittag-Leffler functions of fractional calculus, we propose a flexible and compact functional form.
  arXiv  Detail & Related papers  (2023-02-21T21:20:59Z)
• Uniform Approximation with Quadratic Neural Networks [0.0]
  We show that deep neural networks with ReQU activation can approximate any function within the (R)-H"older-regular functions. Results can be straightforwardly generalized to any Rectified Power Unit (RePU) activation function of the form (max(0,x)p) for (pgeq 2)
  arXiv  Detail & Related papers  (2022-01-11T02:26:55Z)
• An Investigation of Potential Function Designs for Neural CRF [75.79555356970344]
  In this paper, we investigate a series of increasingly expressive potential functions for neural CRF models. Our experiments show that the decomposed quadrilinear potential function based on the vector representations of two neighboring labels and two neighboring words consistently achieves the best performance.
  arXiv  Detail & Related papers  (2020-11-11T07:32:18Z)
• UNIPoint: Universally Approximating Point Processes Intensities [125.08205865536577]
  We provide a proof that a class of learnable functions can universally approximate any valid intensity function. We implement UNIPoint, a novel neural point process model, using recurrent neural networks to parameterise sums of basis function upon each event.
  arXiv  Detail & Related papers  (2020-07-28T09:31:56Z)
• Space of Functions Computed by Deep-Layered Machines [74.13735716675987]
  We study the space of functions computed by random-layered machines, including deep neural networks and Boolean circuits. Investigating the distribution of Boolean functions computed on the recurrent and layer-dependent architectures, we find that it is the same in both models.
  arXiv  Detail & Related papers  (2020-04-19T18:31:03Z)
• Invariant Feature Coding using Tensor Product Representation [75.62232699377877]
  We prove that the group-invariant feature vector contains sufficient discriminative information when learning a linear classifier. A novel feature model that explicitly consider group action is proposed for principal component analysis and k-means clustering.
  arXiv  Detail & Related papers  (2019-06-05T07:15:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.