Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks

• URL: http://arxiv.org/abs/2103.17174v1
• Date: Wed, 31 Mar 2021 15:50:44 GMT
• Title: Using activation histograms to bound the number of affine regions in ReLU feed-forward neural networks
• Abstract summary: Several current bounds on the maximal number of affine regions of a ReLU feed-forward neural network are special cases. We analyze and partially solve a problem in algebraic topology the solution of which would fully exploit this framework.
• Score: 3.04585143845864
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Several current bounds on the maximal number of affine regions of a ReLU feed-forward neural network are special cases of the framework [1] which relies on layer-wise activation histogram bounds. We analyze and partially solve a problem in algebraic topology the solution of which would fully exploit this framework. Our partial solution already induces slightly tighter bounds and suggests insight in how parameter initialization methods can affect the number of regions. Furthermore, we extend the framework to allow the composition of subnetwork instead of layer-wise activation histogram bounds to reduce the number of required compositions which negatively affect the tightness of the resulting bound.

Related papers

• Polyhedral Geometry of Time-to-First-Spike Neural Networks [28.641313300035975]
  We study the expressivity of spiking neural networks, which provide a framework for asynchronous, event-driven computation.<n>We consider the time-to-first-spike model in a setting for which the input-output map is continuous and piecewise linear.<n>We first show that each neuron's firing time admits a maxout-like representation with exponentially many, highly constrained affine pieces.<n>We then formalize causal regions as polyhedral regions with fixed causal sets and derive upper and lower bounds on the maximal number of causal regions in both shallow and multilayer feedforward spiking networks.
  arXiv  Detail & Related papers  (2026-09-10T08:27:38Z)
• Expressivity Saturation: Reduced Affine Region Usage Under Increasing Task Complexity [5.366707531676941]
  Piecewise-affine neural networks implement continuous piecewise-affine maps.<n>We study the gap between theoretical region capacity and the affine regions realized after training.
  arXiv  Detail & Related papers  (2026-06-19T18:55:46Z)
• AB-PINNs: Adaptive-Basis Physics-Informed Neural Networks for Residual-Driven Domain Decomposition [3.0307211002269443]
  We introduce adaptive-basis physics-informed neural networks (AB-PINNs)<n>AB-PINNs dynamically adapt to the intrinsic features of the unknown solution.<n>We present comprehensive numerical results which demonstrate the effectiveness of AB-PINNs at solving a variety of complex partial differential equations.
  arXiv  Detail & Related papers  (2025-10-10T02:13:06Z)
• Topological obstruction to the training of shallow ReLU neural networks [0.0]
  We study the interplay between the geometry of the loss landscape and the optimization trajectories of simple neural networks. This paper reveals the presence of topological obstruction in the loss landscape of shallow ReLU neural networks trained using gradient flow.
  arXiv  Detail & Related papers  (2024-10-18T19:17:48Z)
• Globally Optimal Training of Neural Networks with Threshold Activation Functions [63.03759813952481]
  We study weight decay regularized training problems of deep neural networks with threshold activations. We derive a simplified convex optimization formulation when the dataset can be shattered at a certain layer of the network.
  arXiv  Detail & Related papers  (2023-03-06T18:59:13Z)
• Zonotope Domains for Lagrangian Neural Network Verification [102.13346781220383]
  We decompose the problem of verifying a deep neural network into the verification of many 2-layer neural networks. Our technique yields bounds that improve upon both linear programming and Lagrangian-based verification techniques.
  arXiv  Detail & Related papers  (2022-10-14T19:31:39Z)
• Improved Overparametrization Bounds for Global Convergence of Stochastic Gradient Descent for Shallow Neural Networks [1.14219428942199]
  We study the overparametrization bounds required for the global convergence of gradient descent algorithm for a class of one hidden layer feed-forward neural networks.
  arXiv  Detail & Related papers  (2022-01-28T11:30:06Z)
• Mean-field Analysis of Piecewise Linear Solutions for Wide ReLU Networks [83.58049517083138]
  We consider a two-layer ReLU network trained via gradient descent. We show that SGD is biased towards a simple solution. We also provide empirical evidence that knots at locations distinct from the data points might occur.
  arXiv  Detail & Related papers  (2021-11-03T15:14:20Z)
• Limiting fluctuation and trajectorial stability of multilayer neural networks with mean field training [3.553493344868413]
  We study the fluctuation in the case of multilayer networks at any network depth. We demonstrate through the framework the complex interaction among neurons in this second-order MF limit. A limit theorem is proven to relate this limit to the fluctuation of large-width networks.
  arXiv  Detail & Related papers  (2021-10-29T17:58:09Z)
• Sharp Lower Bounds on the Approximation Rate of Shallow Neural Networks [0.0]
  We prove sharp lower bounds on the approximation rates for shallow neural networks. These lower bounds apply to both sigmoidal activation functions with bounded variation and to activation functions which are a power of the ReLU.
  arXiv  Detail & Related papers  (2021-06-28T22:01:42Z)
• Hierarchical Verification for Adversarial Robustness [89.30150585592648]
  We introduce a new framework for the exact point-wise $ell_p$ robustness verification problem. LayerCert exploits the layer-wise geometric structure of deep feed-forward networks with rectified linear activations (ReLU networks) We show that LayerCert provably reduces the number and size of the convex programs that one needs to solve compared to GeoCert.
  arXiv  Detail & Related papers  (2020-07-23T07:03:05Z)
• Generalization bound of globally optimal non-convex neural network training: Transportation map estimation by infinite dimensional Langevin dynamics [50.83356836818667]
  We introduce a new theoretical framework to analyze deep learning optimization with connection to its generalization error. Existing frameworks such as mean field theory and neural tangent kernel theory for neural network optimization analysis typically require taking limit of infinite width of the network to show its global convergence.
  arXiv  Detail & Related papers  (2020-07-11T18:19:50Z)
• On Sparsity in Overparametrised Shallow ReLU Networks [42.33056643582297]
  We study the ability of different regularisation strategies to capture solutions requiring only a finite amount of neurons, even on the infinitely wide regime. We establish that both schemes are minimised by functions having only a finite number of neurons, irrespective of the amount of overparametrisation.
  arXiv  Detail & Related papers  (2020-06-18T01:35:26Z)
• Local Propagation in Constraint-based Neural Network [77.37829055999238]
  We study a constraint-based representation of neural network architectures. We investigate a simple optimization procedure that is well suited to fulfil the so-called architectural constraints.
  arXiv  Detail & Related papers  (2020-02-18T16:47:38Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.