Backpropagation-free Training of Deep Physical Neural Networks
- URL: http://arxiv.org/abs/2304.11042v3
- Date: Mon, 12 Jun 2023 18:24:02 GMT
- Title: Backpropagation-free Training of Deep Physical Neural Networks
- Authors: Ali Momeni, Babak Rahmani, Matthieu Mallejac, Philipp Del Hougne, and
Romain Fleury
- Abstract summary: We propose a simple deep neural network architecture augmented by a biologically plausible learning algorithm, referred to as "model-free forward-forward training"
We show that our method outperforms state-of-the-art hardware-aware training methods by improving training speed, decreasing digital computations, and reducing power consumption in physical systems.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent years have witnessed the outstanding success of deep learning in
various fields such as vision and natural language processing. This success is
largely indebted to the massive size of deep learning models that is expected
to increase unceasingly. This growth of the deep learning models is accompanied
by issues related to their considerable energy consumption, both during the
training and inference phases, as well as their scalability. Although a number
of work based on unconventional physical systems have been proposed which
addresses the issue of energy efficiency in the inference phase, efficient
training of deep learning models has remained unaddressed. So far, training of
digital deep learning models mainly relies on backpropagation, which is not
suitable for physical implementation as it requires perfect knowledge of the
computation performed in the so-called forward pass of the neural network.
Here, we tackle this issue by proposing a simple deep neural network
architecture augmented by a biologically plausible learning algorithm, referred
to as "model-free forward-forward training". The proposed architecture enables
training deep physical neural networks consisting of layers of physical
nonlinear systems, without requiring detailed knowledge of the nonlinear
physical layers' properties. We show that our method outperforms
state-of-the-art hardware-aware training methods by improving training speed,
decreasing digital computations, and reducing power consumption in physical
systems. We demonstrate the adaptability of the proposed method, even in
systems exposed to dynamic or unpredictable external perturbations. To showcase
the universality of our approach, we train diverse wave-based physical neural
networks that vary in the underlying wave phenomenon and the type of
non-linearity they use, to perform vowel and image classification tasks
experimentally.
Related papers
- Mechanistic Neural Networks for Scientific Machine Learning [58.99592521721158]
We present Mechanistic Neural Networks, a neural network design for machine learning applications in the sciences.
It incorporates a new Mechanistic Block in standard architectures to explicitly learn governing differential equations as representations.
Central to our approach is a novel Relaxed Linear Programming solver (NeuRLP) inspired by a technique that reduces solving linear ODEs to solving linear programs.
arXiv Detail & Related papers (2024-02-20T15:23:24Z) - Optimising network interactions through device agnostic models [2.538490265556881]
Physically implemented neural networks hold the potential to achieve the performance of deep learning models by exploiting the innate physical properties of devices as computational tools.
We formulate a universal framework to optimise interactions with dynamic physical systems in a fully data-driven fashion.
Our work demonstrates the framework's applicability through simulations and physical implementations of interacting dynamic devices, while highlighting the importance of accurately capturing systemity for the successful deployment of a physically defined neural network.
arXiv Detail & Related papers (2024-01-14T22:46:53Z) - Computation-efficient Deep Learning for Computer Vision: A Survey [121.84121397440337]
Deep learning models have reached or even exceeded human-level performance in a range of visual perception tasks.
Deep learning models usually demand significant computational resources, leading to impractical power consumption, latency, or carbon emissions in real-world scenarios.
New research focus is computationally efficient deep learning, which strives to achieve satisfactory performance while minimizing the computational cost during inference.
arXiv Detail & Related papers (2023-08-27T03:55:28Z) - Physics-informed ConvNet: Learning Physical Field from a Shallow Neural
Network [0.180476943513092]
Modelling and forecasting multi-physical systems remain a challenge due to unavoidable data scarcity and noise.
New framework named physics-informed convolutional network (PICN) is recommended from a CNN perspective.
PICN may become an alternative neural network solver in physics-informed machine learning.
arXiv Detail & Related papers (2022-01-26T14:35:58Z) - Constructing Neural Network-Based Models for Simulating Dynamical
Systems [59.0861954179401]
Data-driven modeling is an alternative paradigm that seeks to learn an approximation of the dynamics of a system using observations of the true system.
This paper provides a survey of the different ways to construct models of dynamical systems using neural networks.
In addition to the basic overview, we review the related literature and outline the most significant challenges from numerical simulations that this modeling paradigm must overcome.
arXiv Detail & Related papers (2021-11-02T10:51:42Z) - Dynamic Neural Diversification: Path to Computationally Sustainable
Neural Networks [68.8204255655161]
Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks.
We explore the diversity of the neurons within the hidden layer during the learning process.
We analyze how the diversity of the neurons affects predictions of the model.
arXiv Detail & Related papers (2021-09-20T15:12:16Z) - Deep physical neural networks enabled by a backpropagation algorithm for
arbitrary physical systems [3.7785805908699803]
We propose a radical alternative for implementing deep neural network models: Physical Neural Networks.
We introduce a hybrid physical-digital algorithm called Physics-Aware Training to efficiently train sequences of controllable physical systems to act as deep neural networks.
arXiv Detail & Related papers (2021-04-27T18:00:02Z) - A deep learning theory for neural networks grounded in physics [2.132096006921048]
We argue that building large, fast and efficient neural networks on neuromorphic architectures requires rethinking the algorithms to implement and train them.
Our framework applies to a very broad class of models, namely systems whose state or dynamics are described by variational equations.
arXiv Detail & Related papers (2021-03-18T02:12:48Z) - Learning Contact Dynamics using Physically Structured Neural Networks [81.73947303886753]
We use connections between deep neural networks and differential equations to design a family of deep network architectures for representing contact dynamics between objects.
We show that these networks can learn discontinuous contact events in a data-efficient manner from noisy observations.
Our results indicate that an idealised form of touch feedback is a key component of making this learning problem tractable.
arXiv Detail & Related papers (2021-02-22T17:33:51Z) - Physical deep learning based on optimal control of dynamical systems [0.0]
In this study, we perform pattern recognition based on the optimal control of continuous-time dynamical systems.
As a key example, we apply the dynamics-based recognition approach to an optoelectronic delay system.
This is in contrast to conventional multilayer neural networks, which require a large number of weight parameters to be trained.
arXiv Detail & Related papers (2020-12-16T06:38:01Z) - The large learning rate phase of deep learning: the catapult mechanism [50.23041928811575]
We present a class of neural networks with solvable training dynamics.
We find good agreement between our model's predictions and training dynamics in realistic deep learning settings.
We believe our results shed light on characteristics of models trained at different learning rates.
arXiv Detail & Related papers (2020-03-04T17:52:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.