A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation

• URL: http://arxiv.org/abs/2103.05636v1
• Date: Tue, 9 Mar 2021 02:07:39 GMT
• Title: A Gradient Estimator for Time-Varying Electrical Networks with Non-Linear Dissipation
• Authors: Jack Kendall
• Abstract summary: We use electrical circuit theory to construct a Lagrangian capable of describing deep, directed neural networks. We derive an estimator for the gradient of the physical parameters of the network, such as synapse conductances. We conclude by suggesting methods for extending these results to networks of biologically plausible neurons.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We propose a method for extending the technique of equilibrium propagation for estimating gradients in fixed-point neural networks to the more general setting of directed, time-varying neural networks by modeling them as electrical circuits. We use electrical circuit theory to construct a Lagrangian capable of describing deep, directed neural networks modeled using nonlinear capacitors and inductors, linear resistors and sources, and a special class of nonlinear dissipative elements called fractional memristors. We then derive an estimator for the gradient of the physical parameters of the network, such as synapse conductances, with respect to an arbitrary loss function. This estimator is entirely local, in that it only depends on information locally available to each synapse. We conclude by suggesting methods for extending these results to networks of biologically plausible neurons, e.g. Hodgkin-Huxley neurons.

Related papers

• Speed Limits for Deep Learning [67.69149326107103]
  Recent advancement in thermodynamics allows bounding the speed at which one can go from the initial weight distribution to the final distribution of the fully trained network. We provide analytical expressions for these speed limits for linear and linearizable neural networks. Remarkably, given some plausible scaling assumptions on the NTK spectra and spectral decomposition of the labels -- learning is optimal in a scaling sense.
  arXiv  Detail & Related papers  (2023-07-27T06:59:46Z)
• Addressing caveats of neural persistence with deep graph persistence [54.424983583720675]
  We find that the variance of network weights and spatial concentration of large weights are the main factors that impact neural persistence. We propose an extension of the filtration underlying neural persistence to the whole neural network instead of single layers. This yields our deep graph persistence measure, which implicitly incorporates persistent paths through the network and alleviates variance-related issues.
  arXiv  Detail & Related papers  (2023-07-20T13:34:11Z)
• Learning fixed points of recurrent neural networks by reparameterizing the network model [0.0]
  In computational neuroscience, fixed points of recurrent neural networks are commonly used to model neural responses to static or slowly changing stimuli. A natural approach is to use gradient descent on the Euclidean space of synaptic weights. We show that this approach can lead to poor learning performance due to singularities that arise in the loss surface.
  arXiv  Detail & Related papers  (2023-07-13T13:09:11Z)
• Gradient Descent in Neural Networks as Sequential Learning in RKBS [63.011641517977644]
  We construct an exact power-series representation of the neural network in a finite neighborhood of the initial weights. We prove that, regardless of width, the training sequence produced by gradient descent can be exactly replicated by regularized sequential learning.
  arXiv  Detail & Related papers  (2023-02-01T03:18:07Z)
• Infinite-width limit of deep linear neural networks [0.0]
  We study the infinite-width limit of deep linear neural networks with random parameters. We show that the linear predictors converge at an exponential rate to the minimal $ell$-norm minimizer of the risk.
  arXiv  Detail & Related papers  (2022-11-29T18:37:14Z)
• Spiking neural network for nonlinear regression [68.8204255655161]
  Spiking neural networks carry the potential for a massive reduction in memory and energy consumption. They introduce temporal and neuronal sparsity, which can be exploited by next-generation neuromorphic hardware. A framework for regression using spiking neural networks is proposed.
  arXiv  Detail & Related papers  (2022-10-06T13:04:45Z)
• Training End-to-End Analog Neural Networks with Equilibrium Propagation [64.0476282000118]
  We introduce a principled method to train end-to-end analog neural networks by gradient descent. We show mathematically that a class of analog neural networks (called nonlinear resistive networks) are energy-based models. Our work can guide the development of a new generation of ultra-fast, compact and low-power neural networks supporting on-chip learning.
  arXiv  Detail & Related papers  (2020-06-02T23:38:35Z)
• Neural Network Solutions to Differential Equations in Non-Convex Domains: Solving the Electric Field in the Slit-Well Microfluidic Device [1.7188280334580193]
  The neural network method is used to approximate the electric potential and corresponding electric field in a slit-well microfluidic device. metrics, deep neural networks significantly outperform shallow neural networks.
  arXiv  Detail & Related papers  (2020-04-25T21:20:03Z)
• A Generalized Neural Tangent Kernel Analysis for Two-layer Neural Networks [87.23360438947114]
  We show that noisy gradient descent with weight decay can still exhibit a " Kernel-like" behavior. This implies that the training loss converges linearly up to a certain accuracy. We also establish a novel generalization error bound for two-layer neural networks trained by noisy gradient descent with weight decay.
  arXiv  Detail & Related papers  (2020-02-10T18:56:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.