Training neural networks with end-to-end optical backpropagation
- URL: http://arxiv.org/abs/2308.05226v1
- Date: Wed, 9 Aug 2023 21:11:26 GMT
- Title: Training neural networks with end-to-end optical backpropagation
- Authors: James Spall, Xianxin Guo, A. I. Lvovsky
- Abstract summary: We show how to implement backpropagation, an algorithm for training a neural network, using optical processes.
Our approach is adaptable to various analog platforms, materials, and network structures.
It demonstrates the possibility of constructing neural networks entirely reliant on analog optical processes for both training and inference tasks.
- Score: 1.1602089225841632
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Optics is an exciting route for the next generation of computing hardware for
machine learning, promising several orders of magnitude enhancement in both
computational speed and energy efficiency. However, to reach the full capacity
of an optical neural network it is necessary that the computing not only for
the inference, but also for the training be implemented optically. The primary
algorithm for training a neural network is backpropagation, in which the
calculation is performed in the order opposite to the information flow for
inference. While straightforward in a digital computer, optical implementation
of backpropagation has so far remained elusive, particularly because of the
conflicting requirements for the optical element that implements the nonlinear
activation function. In this work, we address this challenge for the first time
with a surprisingly simple and generic scheme. Saturable absorbers are employed
for the role of the activation units, and the required properties are achieved
through a pump-probe process, in which the forward propagating signal acts as
the pump and backward as the probe. Our approach is adaptable to various analog
platforms, materials, and network structures, and it demonstrates the
possibility of constructing neural networks entirely reliant on analog optical
processes for both training and inference tasks.
Related papers
- Genetically programmable optical random neural networks [0.0]
We demonstrate a genetically programmable yet simple optical neural network to achieve high performances with optical random projection.
By genetically programming the orientation of the scattering medium which acts as a random projection kernel, our novel technique finds an optimum kernel and improves its initial test accuracies 7-22%.
Our optical computing method presents a promising approach to achieve high performance in optical neural networks with a simple and scalable design.
arXiv Detail & Related papers (2024-03-19T06:55:59Z) - Contrastive-Signal-Dependent Plasticity: Forward-Forward Learning of
Spiking Neural Systems [73.18020682258606]
We develop a neuro-mimetic architecture, composed of spiking neuronal units, where individual layers of neurons operate in parallel.
We propose an event-based generalization of forward-forward learning, which we call contrastive-signal-dependent plasticity (CSDP)
Our experimental results on several pattern datasets demonstrate that the CSDP process works well for training a dynamic recurrent spiking network capable of both classification and reconstruction.
arXiv Detail & Related papers (2023-03-30T02:40:28Z) - The Predictive Forward-Forward Algorithm [79.07468367923619]
We propose the predictive forward-forward (PFF) algorithm for conducting credit assignment in neural systems.
We design a novel, dynamic recurrent neural system that learns a directed generative circuit jointly and simultaneously with a representation circuit.
PFF efficiently learns to propagate learning signals and updates synapses with forward passes only.
arXiv Detail & Related papers (2023-01-04T05:34:48Z) - Realization of a quantum neural network using repeat-until-success
circuits in a superconducting quantum processor [0.0]
In this paper, we use repeat-until-success circuits enabled by real-time control-flow feedback to realize quantum neurons with non-linear activation functions.
As an example, we construct a minimal feedforward quantum neural network capable of learning all 2-to-1-bit Boolean functions.
This model is shown to perform non-linear classification and effectively learns from multiple copies of a single training state consisting of the maximal superposition of all inputs.
arXiv Detail & Related papers (2022-12-21T03:26:32Z) - Simple initialization and parametrization of sinusoidal networks via
their kernel bandwidth [92.25666446274188]
sinusoidal neural networks with activations have been proposed as an alternative to networks with traditional activation functions.
We first propose a simplified version of such sinusoidal neural networks, which allows both for easier practical implementation and simpler theoretical analysis.
We then analyze the behavior of these networks from the neural tangent kernel perspective and demonstrate that their kernel approximates a low-pass filter with an adjustable bandwidth.
arXiv Detail & Related papers (2022-11-26T07:41:48Z) - Physical Deep Learning with Biologically Plausible Training Method [2.5608506499175094]
We present physical deep learning by extending a biologically plausible training algorithm called direct feedback alignment.
We can emulate and accelerate the computation for this training on a simple and scalable physical system.
Our results provide practical solutions for the training and acceleration of neuromorphic computation.
arXiv Detail & Related papers (2022-04-01T05:46:16Z) - Scale-, shift- and rotation-invariant diffractive optical networks [0.0]
Diffractive Deep Neural Networks (D2NNs) harness light-matter interaction over a series of trainable surfaces to compute a desired statistical inference task.
Here, we demonstrate a new training strategy for diffractive networks that introduces input object translation, rotation and/or scaling during the training phase.
This training strategy successfully guides the evolution of the diffractive optical network design towards a solution that is scale-, shift- and rotation-invariant.
arXiv Detail & Related papers (2020-10-24T02:18:39Z) - Rapid characterisation of linear-optical networks via PhaseLift [51.03305009278831]
Integrated photonics offers great phase-stability and can rely on the large scale manufacturability provided by the semiconductor industry.
New devices, based on such optical circuits, hold the promise of faster and energy-efficient computations in machine learning applications.
We present a novel technique to reconstruct the transfer matrix of linear optical networks.
arXiv Detail & Related papers (2020-10-01T16:04:22Z) - Training End-to-End Analog Neural Networks with Equilibrium Propagation [64.0476282000118]
We introduce a principled method to train end-to-end analog neural networks by gradient descent.
We show mathematically that a class of analog neural networks (called nonlinear resistive networks) are energy-based models.
Our work can guide the development of a new generation of ultra-fast, compact and low-power neural networks supporting on-chip learning.
arXiv Detail & Related papers (2020-06-02T23:38:35Z) - Light-in-the-loop: using a photonics co-processor for scalable training
of neural networks [21.153688679957337]
We present the first optical co-processor able to accelerate the training phase of digitally-implemented neural networks.
We demonstrate its use to train a neural network for handwritten digits recognition.
arXiv Detail & Related papers (2020-06-02T09:19:45Z) - Spiking Neural Networks Hardware Implementations and Challenges: a
Survey [53.429871539789445]
Spiking Neural Networks are cognitive algorithms mimicking neuron and synapse operational principles.
We present the state of the art of hardware implementations of spiking neural networks.
We discuss the strategies employed to leverage the characteristics of these event-driven algorithms at the hardware level.
arXiv Detail & Related papers (2020-05-04T13:24:00Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.