Backpropagation-free Training of Deep Physical Neural Networks
- URL: http://arxiv.org/abs/2304.11042v3
- Date: Mon, 12 Jun 2023 18:24:02 GMT
- Title: Backpropagation-free Training of Deep Physical Neural Networks
- Authors: Ali Momeni, Babak Rahmani, Matthieu Mallejac, Philipp Del Hougne, and
Romain Fleury
- Abstract summary: We propose a simple deep neural network architecture augmented by a biologically plausible learning algorithm, referred to as "model-free forward-forward training"
We show that our method outperforms state-of-the-art hardware-aware training methods by improving training speed, decreasing digital computations, and reducing power consumption in physical systems.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent years have witnessed the outstanding success of deep learning in
various fields such as vision and natural language processing. This success is
largely indebted to the massive size of deep learning models that is expected
to increase unceasingly. This growth of the deep learning models is accompanied
by issues related to their considerable energy consumption, both during the
training and inference phases, as well as their scalability. Although a number
of work based on unconventional physical systems have been proposed which
addresses the issue of energy efficiency in the inference phase, efficient
training of deep learning models has remained unaddressed. So far, training of
digital deep learning models mainly relies on backpropagation, which is not
suitable for physical implementation as it requires perfect knowledge of the
computation performed in the so-called forward pass of the neural network.
Here, we tackle this issue by proposing a simple deep neural network
architecture augmented by a biologically plausible learning algorithm, referred
to as "model-free forward-forward training". The proposed architecture enables
training deep physical neural networks consisting of layers of physical
nonlinear systems, without requiring detailed knowledge of the nonlinear
physical layers' properties. We show that our method outperforms
state-of-the-art hardware-aware training methods by improving training speed,
decreasing digital computations, and reducing power consumption in physical
systems. We demonstrate the adaptability of the proposed method, even in
systems exposed to dynamic or unpredictable external perturbations. To showcase
the universality of our approach, we train diverse wave-based physical neural
networks that vary in the underlying wave phenomenon and the type of
non-linearity they use, to perform vowel and image classification tasks
experimentally.
Related papers
- Contrastive Learning in Memristor-based Neuromorphic Systems [55.11642177631929]
Spiking neural networks have become an important family of neuron-based models that sidestep many of the key limitations facing modern-day backpropagation-trained deep networks.
In this work, we design and investigate a proof-of-concept instantiation of contrastive-signal-dependent plasticity (CSDP), a neuromorphic form of forward-forward-based, backpropagation-free learning.
arXiv Detail & Related papers (2024-09-17T04:48:45Z) - Mechanistic Neural Networks for Scientific Machine Learning [58.99592521721158]
We present Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences.
It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations.
Central to our approach is a novel Relaxed Linear Programming solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs.
arXiv Detail & Related papers (2024-02-20T15:23:24Z) - Physics guided neural networks for modelling of non-linear dynamics [0.0]
This work demonstrates that injection of partially known information at an intermediate layer in a deep neural network can improve model accuracy, reduce model uncertainty, and yield improved convergence during the training.
The value of these physics-guided neural networks has been demonstrated by learning the dynamics of a wide variety of nonlinear dynamical systems represented by five well-known equations in nonlinear systems theory.
arXiv Detail & Related papers (2022-05-13T19:06:36Z) - Physics-informed ConvNet: Learning Physical Field from a Shallow Neural
Network [0.180476943513092]
Modelling and forecasting multi-physical systems remain a challenge due to unavoidable data scarcity and noise.
New framework named physics-informed convolutional network (PICN) is recommended from a CNN perspective.
PICN may become an alternative neural network solver in physics-informed machine learning.
arXiv Detail & Related papers (2022-01-26T14:35:58Z) - Constructing Neural Network-Based Models for Simulating Dynamical
Systems [59.0861954179401]
Data-driven modeling is an alternative paradigm that seeks to learn an approximation of the dynamics of a system using observations of the true system.
This paper provides a survey of the different ways to construct models of dynamical systems using neural networks.
In addition to the basic overview, we review the related literature and outline the most significant challenges from numerical simulations that this modeling paradigm must overcome.
arXiv Detail & Related papers (2021-11-02T10:51:42Z) - Dynamic Neural Diversification: Path to Computationally Sustainable
Neural Networks [68.8204255655161]
Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks.
We explore the diversity of the neurons within the hidden layer during the learning process.
We analyze how the diversity of the neurons affects predictions of the model.
arXiv Detail & Related papers (2021-09-20T15:12:16Z) - Deep physical neural networks enabled by a backpropagation algorithm for
arbitrary physical systems [3.7785805908699803]
We propose a radical alternative for implementing deep neural network models: Physical Neural Networks.
We introduce a hybrid physical-digital algorithm called Physics-Aware Training to efficiently train sequences of controllable physical systems to act as deep neural networks.
arXiv Detail & Related papers (2021-04-27T18:00:02Z) - A deep learning theory for neural networks grounded in physics [2.132096006921048]
We argue that building large, fast and efficient neural networks on neuromorphic architectures requires rethinking the algorithms to implement and train them.
Our framework applies to a very broad class of models, namely systems whose state or dynamics are described by variational equations.
arXiv Detail & Related papers (2021-03-18T02:12:48Z) - Learning Contact Dynamics using Physically Structured Neural Networks [81.73947303886753]
We use connections between deep neural networks and differential equations to design a family of deep network architectures for representing contact dynamics between objects.
We show that these networks can learn discontinuous contact events in a data-efficient manner from noisy observations.
Our results indicate that an idealised form of touch feedback is a key component of making this learning problem tractable.
arXiv Detail & Related papers (2021-02-22T17:33:51Z) - Physical deep learning based on optimal control of dynamical systems [0.0]
In this study, we perform pattern recognition based on the optimal control of continuous-time dynamical systems.
As a key example, we apply the dynamics-based recognition approach to an optoelectronic delay system.
This is in contrast to conventional multilayer neural networks, which require a large number of weight parameters to be trained.
arXiv Detail & Related papers (2020-12-16T06:38:01Z) - The large learning rate phase of deep learning: the catapult mechanism [50.23041928811575]
We present a class of neural networks with solvable training dynamics.
We find good agreement between our model's predictions and training dynamics in realistic deep learning settings.
We believe our results shed light on characteristics of models trained at different learning rates.
arXiv Detail & Related papers (2020-03-04T17:52:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.