Deep Reinforcement Learning with Population-Coded Spiking Neural Network
for Continuous Control
- URL: http://arxiv.org/abs/2010.09635v1
- Date: Mon, 19 Oct 2020 16:20:45 GMT
- Title: Deep Reinforcement Learning with Population-Coded Spiking Neural Network
for Continuous Control
- Authors: Guangzhi Tang, Neelesh Kumar, Raymond Yoo, Konstantinos P. Michmizos
- Abstract summary: We propose a population-coded spiking actor network (PopSAN) trained in conjunction with a deep critic network using deep reinforcement learning (DRL)
We deployed the trained PopSAN on Intel's Loihi neuromorphic chip and benchmarked our method against the mainstream DRL algorithms for continuous control.
Our results support the efficiency of neuromorphic controllers and suggest our hybrid RL as an alternative to deep learning, when both energy-efficiency and robustness are important.
- Score: 0.0
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: The energy-efficient control of mobile robots is crucial as the complexity of
their real-world applications increasingly involves high-dimensional
observation and action spaces, which cannot be offset by limited on-board
resources. An emerging non-Von Neumann model of intelligence, where spiking
neural networks (SNNs) are run on neuromorphic processors, is regarded as an
energy-efficient and robust alternative to the state-of-the-art real-time
robotic controllers for low dimensional control tasks. The challenge now for
this new computing paradigm is to scale so that it can keep up with real-world
tasks. To do so, SNNs need to overcome the inherent limitations of their
training, namely the limited ability of their spiking neurons to represent
information and the lack of effective learning algorithms. Here, we propose a
population-coded spiking actor network (PopSAN) trained in conjunction with a
deep critic network using deep reinforcement learning (DRL). The population
coding scheme dramatically increased the representation capacity of the network
and the hybrid learning combined the training advantages of deep networks with
the energy-efficient inference of spiking networks. To show the general
applicability of our approach, we integrated it with a spectrum of both
on-policy and off-policy DRL algorithms. We deployed the trained PopSAN on
Intel's Loihi neuromorphic chip and benchmarked our method against the
mainstream DRL algorithms for continuous control. To allow for a fair
comparison among all methods, we validated them on OpenAI gym tasks. Our
Loihi-run PopSAN consumed 140 times less energy per inference when compared
against the deep actor network on Jetson TX2, and had the same level of
performance. Our results support the efficiency of neuromorphic controllers and
suggest our hybrid RL as an alternative to deep learning, when both
energy-efficiency and robustness are important.
Related papers
- Fully Spiking Actor Network with Intra-layer Connections for
Reinforcement Learning [51.386945803485084]
We focus on the task where the agent needs to learn multi-dimensional deterministic policies to control.
Most existing spike-based RL methods take the firing rate as the output of SNNs, and convert it to represent continuous action space (i.e., the deterministic policy) through a fully-connected layer.
To develop a fully spiking actor network without any floating-point matrix operations, we draw inspiration from the non-spiking interneurons found in insects.
arXiv Detail & Related papers (2024-01-09T07:31:34Z) - Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency.
We present an IPU-optimized release of our custom SNN Python package, snnTorch.
arXiv Detail & Related papers (2022-11-19T15:44:08Z) - Deep Reinforcement Learning with Spiking Q-learning [51.386945803485084]
spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption.
It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (RL)
arXiv Detail & Related papers (2022-01-21T16:42:11Z) - Population-coding and Dynamic-neurons improved Spiking Actor Network for
Reinforcement Learning [10.957578424267757]
Spiking Neural Network (SNN) contains a diverse population of spiking neurons, making it naturally powerful on state representation with spatial and temporal information.
We propose a Population-coding and Dynamic-neurons improved Spiking Actor Network (PDSAN) for efficient state representation from two different scales.
Our TD3-PDSAN model achieves better performance than state-of-the-art models on four OpenAI gym benchmark tasks.
arXiv Detail & Related papers (2021-06-15T03:14:41Z) - SpikeDyn: A Framework for Energy-Efficient Spiking Neural Networks with
Continual and Unsupervised Learning Capabilities in Dynamic Environments [14.727296040550392]
Spiking Neural Networks (SNNs) bear the potential of efficient unsupervised and continual learning capabilities because of their biological plausibility.
We propose SpikeDyn, a framework for energy-efficient SNNs with continual and unsupervised learning capabilities in dynamic environments.
arXiv Detail & Related papers (2021-02-28T08:26:23Z) - Optimizing Memory Placement using Evolutionary Graph Reinforcement
Learning [56.83172249278467]
We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces.
We train and validate our approach directly on the Intel NNP-I chip for inference.
We additionally achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.
arXiv Detail & Related papers (2020-07-14T18:50:12Z) - Progressive Tandem Learning for Pattern Recognition with Deep Spiking
Neural Networks [80.15411508088522]
Spiking neural networks (SNNs) have shown advantages over traditional artificial neural networks (ANNs) for low latency and high computational efficiency.
We propose a novel ANN-to-SNN conversion and layer-wise learning framework for rapid and efficient pattern recognition.
arXiv Detail & Related papers (2020-07-02T15:38:44Z) - Reinforcement co-Learning of Deep and Spiking Neural Networks for
Energy-Efficient Mapless Navigation with Neuromorphic Hardware [0.0]
We propose a neuromorphic approach that combines the energy-efficiency of spiking neural networks with the optimality of deep reinforcement learning (DRL)
Our framework consists of a spiking actor network (SAN) and a deep critic network, where the two networks were trained jointly using gradient descent.
To evaluate our approach, we deployed the trained SAN on Intel's Loihi neuromorphic processor.
arXiv Detail & Related papers (2020-03-02T19:39:16Z) - Deep Learning for Ultra-Reliable and Low-Latency Communications in 6G
Networks [84.2155885234293]
We first summarize how to apply data-driven supervised deep learning and deep reinforcement learning in URLLC.
To address these open problems, we develop a multi-level architecture that enables device intelligence, edge intelligence, and cloud intelligence for URLLC.
arXiv Detail & Related papers (2020-02-22T14:38:11Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.