Fixed width treelike neural networks capacity analysis -- generic activations

• URL: http://arxiv.org/abs/2402.05696v1
• Date: Thu, 8 Feb 2024 14:19:29 GMT
• Title: Fixed width treelike neural networks capacity analysis -- generic activations
• Abstract summary: We consider the capacity of emphtreelike committee machines (TCM) neural networks. We show that the frameworks of citeStojnictcmspnncaprdt23,Stojnictcmspnncapliftedrdt23 are sufficiently powerful to enable handling of such scenarios as well.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We consider the capacity of \emph{treelike committee machines} (TCM) neural networks. Relying on Random Duality Theory (RDT), \cite{Stojnictcmspnncaprdt23} recently introduced a generic framework for their capacity analysis. An upgrade based on the so-called \emph{partially lifted} RDT (pl RDT) was then presented in \cite{Stojnictcmspnncapliftedrdt23}. Both lines of work focused on the networks with the most typical, \emph{sign}, activations. Here, on the other hand, we focus on networks with other, more general, types of activations and show that the frameworks of \cite{Stojnictcmspnncaprdt23,Stojnictcmspnncapliftedrdt23} are sufficiently powerful to enable handling of such scenarios as well. In addition to the standard \emph{linear} activations, we uncover that particularly convenient results can be obtained for two very commonly used activations, namely, the \emph{quadratic} and \emph{rectified linear unit (ReLU)} ones. In more concrete terms, for each of these activations, we obtain both the RDT and pl RDT based memory capacities upper bound characterization for \emph{any} given (even) number of the hidden layer neurons, $d$. In the process, we also uncover the following two, rather remarkable, facts: 1) contrary to the common wisdom, both sets of results show that the bounding capacity decreases for large $d$ (the width of the hidden layer) while converging to a constant value; and 2) the maximum bounding capacity is achieved for the networks with precisely \textbf{\emph{two}} hidden layer neurons! Moreover, the large $d$ converging values are observed to be in excellent agrement with the statistical physics replica theory based predictions.

Related papers

• Sparse by Rule: Probability-Based N:M Pruning for Spiking Neural Networks [66.61171793101872]
  Spiking neural networks (SNNs) promise energy-efficient intelligence via event-driven, sparse computation, but deeper architectures inflate parameters and computational cost, hindering their edge deployment.<n>Recent progress in SNN pruning helps alleviate this burden, yet existing efforts fall into only two families: emphunstructured pruning, which attains high sparsity but is difficult to accelerate on general hardware, and emphstructured pruning, which eases deployment but lack flexibility and often degrades accuracy at matched sparsity.<n>We introduce bfSpikeNM, the first SNN
  arXiv  Detail & Related papers  (2025-11-15T08:23:20Z)
• Exact capacity of the \emph{wide} hidden layer treelike neural networks with generic activations [0.0]
  Random Duality Theory (RDT) and its a emphpartially lifted(pl RDT) variant are powerful tools that can be used for very precise networks capacity analysis. Here, we consider emphwide hidden layer networks and uncover that certain aspects of numerical difficulties faced in citeStojnictcmspnncapdiffactrdt23 miraculously disappear.
  arXiv  Detail & Related papers  (2024-02-08T14:50:07Z)
• \emph{Lifted} RDT based capacity analysis of the 1-hidden layer treelike \emph{sign} perceptrons neural networks [0.0]
  We consider the capabilities of multilayered emphsign perceptrons neural networks (SPNNs) A recent rigorous capacity characterization, obtained in citeStojnictcmspnncaprdt23, demonstrated that adding neurons in a network configuration may indeed be very beneficial.
  arXiv  Detail & Related papers  (2023-12-13T16:19:58Z)
• Capacity of the treelike sign perceptrons neural networks with one hidden layer -- RDT based upper bounds [0.0]
  We study the capacity of emphsign perceptrons neural networks (SPNN) We focus on 1-hidden layer emphtreelike committee machine (TCM) architectures.
  arXiv  Detail & Related papers  (2023-12-13T16:06:18Z)
• Binary perceptrons capacity via fully lifted random duality theory [0.0]
  We study the statistical capacity of the classical binary perceptrons with general thresholds $kappa$. We rely on emphfully lifted random duality theory (fl RDT) established in citeStojnicflrdt23 to create a general framework for studying the perceptrons' capacities.
  arXiv  Detail & Related papers  (2023-11-29T22:22:32Z)
• Kronecker-Factored Approximate Curvature for Modern Neural Network Architectures [85.76673783330334]
  Two different settings of linear weight-sharing layers motivate two flavours of Kronecker-Factored Approximate Curvature (K-FAC) We show they are exact for deep linear networks with weight-sharing in their respective setting. We observe little difference between these two K-FAC variations when using them to train both a graph neural network and a vision transformer.
  arXiv  Detail & Related papers  (2023-11-01T16:37:00Z)
• Memory capacity of two layer neural networks with smooth activations [27.33243506775655]
  We determine the memory capacity of two layer neural networks with $m$ hidden neurons and input dimension $d$. We derive the precise generic rank of the network's Jacobian, which can be written in terms of Hadamard powers. Our approach differs from prior works on memory capacity and holds promise for extending to deeper models.
  arXiv  Detail & Related papers  (2023-08-03T19:31:15Z)
• Improved Convergence Guarantees for Shallow Neural Networks [91.3755431537592]
  We prove convergence of depth 2 neural networks, trained via gradient descent, to a global minimum. Our model has the following features: regression with quadratic loss function, fully connected feedforward architecture, RelU activations, Gaussian data instances, adversarial labels. They strongly suggest that, at least in our model, the convergence phenomenon extends well beyond the NTK regime''
  arXiv  Detail & Related papers  (2022-12-05T14:47:52Z)
• Robust Training and Verification of Implicit Neural Networks: A Non-Euclidean Contractive Approach [64.23331120621118]
  This paper proposes a theoretical and computational framework for training and robustness verification of implicit neural networks. We introduce a related embedded network and show that the embedded network can be used to provide an $ell_infty$-norm box over-approximation of the reachable sets of the original network. We apply our algorithms to train implicit neural networks on the MNIST dataset and compare the robustness of our models with the models trained via existing approaches in the literature.
  arXiv  Detail & Related papers  (2022-08-08T03:13:24Z)
• On the Neural Tangent Kernel Analysis of Randomly Pruned Neural Networks [91.3755431537592]
  We study how random pruning of the weights affects a neural network's neural kernel (NTK) In particular, this work establishes an equivalence of the NTKs between a fully-connected neural network and its randomly pruned version.
  arXiv  Detail & Related papers  (2022-03-27T15:22:19Z)
• Scalable Lipschitz Residual Networks with Convex Potential Flows [120.27516256281359]
  We show that using convex potentials in a residual network gradient flow provides a built-in $1$-Lipschitz transformation. A comprehensive set of experiments on CIFAR-10 demonstrates the scalability of our architecture and the benefit of our approach for $ell$ provable defenses.
  arXiv  Detail & Related papers  (2021-10-25T07:12:53Z)
• ActNN: Reducing Training Memory Footprint via 2-Bit Activation Compressed Training [68.63354877166756]
  ActNN is a memory-efficient training framework that stores randomly quantized activations for back propagation. ActNN reduces the memory footprint of the activation by 12x, and it enables training with a 6.6x to 14x larger batch size.
  arXiv  Detail & Related papers  (2021-04-29T05:50:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.