One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks

• URL: http://arxiv.org/abs/2303.06316v1
• Date: Sat, 11 Mar 2023 05:32:18 GMT
• Title: One Neuron Saved Is One Neuron Earned: On Parametric Efficiency of Quadratic Networks
• Authors: Feng-Lei Fan, Hang-Cheng Dong, Zhongming Wu, Lecheng Ruan, Tieyong Zeng, Yiming Cui, Jing-Xiao Liao
• Abstract summary: We show that quadratic networks enjoy parametric efficiency, thereby confirming that the superior performance of quadratic networks is due to the intrinsic expressive capability. From the perspective of the Barron space, we demonstrate that there exists a functional space whose functions can be approximated by quadratic networks in a dimension-free error.
• Score: 21.5187335186035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Inspired by neuronal diversity in the biological neural system, a plethora of studies proposed to design novel types of artificial neurons and introduce neuronal diversity into artificial neural networks. Recently proposed quadratic neuron, which replaces the inner-product operation in conventional neurons with a quadratic one, have achieved great success in many essential tasks. Despite the promising results of quadratic neurons, there is still an unresolved issue: \textit{Is the superior performance of quadratic networks simply due to the increased parameters or due to the intrinsic expressive capability?} Without clarifying this issue, the performance of quadratic networks is always suspicious. Additionally, resolving this issue is reduced to finding killer applications of quadratic networks. In this paper, with theoretical and empirical studies, we show that quadratic networks enjoy parametric efficiency, thereby confirming that the superior performance of quadratic networks is due to the intrinsic expressive capability. This intrinsic expressive ability comes from that quadratic neurons can easily represent nonlinear interaction, while it is hard for conventional neurons. Theoretically, we derive the approximation efficiency of the quadratic network over conventional ones in terms of real space and manifolds. Moreover, from the perspective of the Barron space, we demonstrate that there exists a functional space whose functions can be approximated by quadratic networks in a dimension-free error, but the approximation error of conventional networks is dependent on dimensions. Empirically, experimental results on synthetic data, classic benchmarks, and real-world applications show that quadratic models broadly enjoy parametric efficiency, and the gain of efficiency depends on the task.

Related papers

• Efficient Vectorized Backpropagation Algorithms for Training Feedforward Networks Composed of Quadratic Neurons [1.6574413179773761]
  This paper presents a solution to the XOR problem with a single quadratic neuron. It shows that any dataset composed of $mathcalC$ bounded clusters can be separated with only a single layer of $mathcalC$ quadratic neurons.
  arXiv  Detail & Related papers  (2023-10-04T15:39:57Z)
• Computational and Storage Efficient Quadratic Neurons for Deep Neural Networks [10.379191500493503]
  Experimental results have demonstrated that the proposed quadratic neuron structure exhibits superior computational and storage efficiency across various tasks. This work introduces an efficient quadratic neuron architecture distinguished by its enhanced utilization of second-order computational information.
  arXiv  Detail & Related papers  (2023-06-10T11:25:31Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Two-argument activation functions learn soft XOR operations like cortical neurons [6.88204255655161]
  We learn canonical activation functions with two input arguments, analogous to basal and apical dendrites. Remarkably, the resultant nonlinearities often produce soft XOR functions. Networks with these nonlinearities learn faster and perform better than conventional ReLU nonlinearities with matched parameter counts.
  arXiv  Detail & Related papers  (2021-10-13T17:06:20Z)
• On Expressivity and Trainability of Quadratic Networks [12.878230964137014]
  quadratic artificial neurons can play an important role in deep learning models. We show that the superior expressivity of a quadratic network over either a conventional network or a conventional network via quadratic activation is not fully elucidated. We propose an effective training strategy referred to as ReLinear to stabilize the training process of a quadratic network.
  arXiv  Detail & Related papers  (2021-10-12T15:33:32Z)
• Going Beyond Linear RL: Sample Efficient Neural Function Approximation [76.57464214864756]
  We study function approximation with two-layer neural networks. Our results significantly improve upon what can be attained with linear (or eluder dimension) methods.
  arXiv  Detail & Related papers  (2021-07-14T03:03:56Z)
• Recognizing and Verifying Mathematical Equations using Multiplicative Differential Neural Units [86.9207811656179]
  We show that memory-augmented neural networks (NNs) can achieve higher-order, memory-augmented extrapolation, stable performance, and faster convergence. Our models achieve a 1.53% average improvement over current state-of-the-art methods in equation verification and achieve a 2.22% Top-1 average accuracy and 2.96% Top-5 average accuracy for equation completion.
  arXiv  Detail & Related papers  (2021-04-07T03:50:11Z)
• And/or trade-off in artificial neurons: impact on adversarial robustness [91.3755431537592]
  Presence of sufficient number of OR-like neurons in a network can lead to classification brittleness and increased vulnerability to adversarial attacks. We define AND-like neurons and propose measures to increase their proportion in the network. Experimental results on the MNIST dataset suggest that our approach holds promise as a direction for further exploration.
  arXiv  Detail & Related papers  (2021-02-15T08:19:05Z)
• Flexible Transmitter Network [84.90891046882213]
  Current neural networks are mostly built upon the MP model, which usually formulates the neuron as executing an activation function on the real-valued weighted aggregation of signals received from other neurons. We propose the Flexible Transmitter (FT) model, a novel bio-plausible neuron model with flexible synaptic plasticity. We present the Flexible Transmitter Network (FTNet), which is built on the most common fully-connected feed-forward architecture.
  arXiv  Detail & Related papers  (2020-04-08T06:55:12Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)
• Avoiding Spurious Local Minima in Deep Quadratic Networks [0.0]
  We characterize the landscape of the mean squared nonlinear error for networks with neural activation functions. We prove that deepized neural networks with quadratic activations benefit from similar landscape properties.
  arXiv  Detail & Related papers  (2019-12-31T22:31:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.