SpikeRL: A Scalable and Energy-efficient Framework for Deep Spiking Reinforcement Learning

• URL: http://arxiv.org/abs/2502.17496v1
• Date: Fri, 21 Feb 2025 05:28:42 GMT
• Title: SpikeRL: A Scalable and Energy-efficient Framework for Deep Spiking Reinforcement Learning
• Authors: Tokey Tahmid, Mark Gates, Piotr Luszczek, Catherine D. Schuman,
• Abstract summary: SpikeRL is a scalable and energy efficient framework for DeepRL-based SNNs for continuous control.<n>Our new SpikeRL implementation is 4.26X faster and 2.25X more energy efficient than state-of-the-art DeepRL-SNN methods.
• Score: 1.6999370482438731
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: In this era of AI revolution, massive investments in large-scale data-driven AI systems demand high-performance computing, consuming tremendous energy and resources. This trend raises new challenges in optimizing sustainability without sacrificing scalability or performance. Among the energy-efficient alternatives of the traditional Von Neumann architecture, neuromorphic computing and its Spiking Neural Networks (SNNs) are a promising choice due to their inherent energy efficiency. However, in some real-world application scenarios such as complex continuous control tasks, SNNs often lack the performance optimizations that traditional artificial neural networks have. Researchers have addressed this by combining SNNs with Deep Reinforcement Learning (DeepRL), yet scalability remains unexplored. In this paper, we extend our previous work on SpikeRL, which is a scalable and energy efficient framework for DeepRL-based SNNs for continuous control. In our initial implementation of SpikeRL framework, we depended on the population encoding from the Population-coded Spiking Actor Network (PopSAN) method for our SNN model and implemented distributed training with Message Passing Interface (MPI) through mpi4py. Also, further optimizing our model training by using mixed-precision for parameter updates. In our new SpikeRL framework, we have implemented our own DeepRL-SNN component with population encoding, and distributed training with PyTorch Distributed package with NCCL backend while still optimizing with mixed precision training. Our new SpikeRL implementation is 4.26X faster and 2.25X more energy efficient than state-of-the-art DeepRL-SNN methods. Our proposed SpikeRL framework demonstrates a truly scalable and sustainable solution for complex continuous control tasks in real-world applications.

Related papers

• Decision SpikeFormer: Spike-Driven Transformer for Decision Making [11.652964678824382]
  offline reinforcement learning (RL) enables policy training solely on pre-collected data, avoiding direct environment interaction. We introduce DSFormer, the first spike-driven transformer model designed to tackle offline RL via sequence modeling. Comprehensive results in the D4RL benchmark show DSFormer's superiority over both SNN and ANN counterparts, achieving 78.4% energy savings.
  arXiv  Detail & Related papers  (2025-04-04T07:42:36Z)
• Achieving Network Resilience through Graph Neural Network-enabled Deep Reinforcement Learning [64.20847540439318]
  Deep reinforcement learning (DRL) has been widely used in many important tasks of communication networks.<n>Some studies have combined graph neural networks (GNNs) with DRL, which use the GNNs to extract unstructured features of the network.<n>This paper explores the solution of combining GNNs with DRL to build a resilient network.
  arXiv  Detail & Related papers  (2025-01-19T15:22:17Z)
• DNN Partitioning, Task Offloading, and Resource Allocation in Dynamic Vehicular Networks: A Lyapunov-Guided Diffusion-Based Reinforcement Learning Approach [49.56404236394601]
  We formulate the problem of joint DNN partitioning, task offloading, and resource allocation in Vehicular Edge Computing. Our objective is to minimize the DNN-based task completion time while guaranteeing the system stability over time. We propose a Multi-Agent Diffusion-based Deep Reinforcement Learning (MAD2RL) algorithm, incorporating the innovative use of diffusion models.
  arXiv  Detail & Related papers  (2024-06-11T06:31:03Z)
• Spyx: A Library for Just-In-Time Compiled Optimization of Spiking Neural Networks [0.08965418284317034]
  Spiking Neural Networks (SNNs) offer to enhance energy efficiency through a reduced and low-power hardware footprint. This paper introduces Spyx, a new and lightweight SNN simulation and optimization library designed in JAX.
  arXiv  Detail & Related papers  (2024-02-29T09:46:44Z)
• LitE-SNN: Designing Lightweight and Efficient Spiking Neural Network through Spatial-Temporal Compressive Network Search and Joint Optimization [48.41286573672824]
  Spiking Neural Networks (SNNs) mimic the information-processing mechanisms of the human brain and are highly energy-efficient. We propose a new approach named LitE-SNN that incorporates both spatial and temporal compression into the automated network design process.
  arXiv  Detail & Related papers  (2024-01-26T05:23:11Z)
• A Low Latency Adaptive Coding Spiking Framework for Deep Reinforcement Learning [27.558298367330053]
  In this paper, we use learnable matrix multiplication to encode and decode spikes, improving the flexibility of the coders. We train the SNNs using the direct training method and use two different structures for online and offline RL algorithms. Experiments have revealed that our method achieves optimal performance with ultra-low latency and excellent energy efficiency.
  arXiv  Detail & Related papers  (2022-11-21T07:26:56Z)
• Intelligence Processing Units Accelerate Neuromorphic Learning [52.952192990802345]
  Spiking neural networks (SNNs) have achieved orders of magnitude improvement in terms of energy consumption and latency. We present an IPU-optimized release of our custom SNN Python package, snnTorch.
  arXiv  Detail & Related papers  (2022-11-19T15:44:08Z)
• Deep Reinforcement Learning with Spiking Q-learning [51.386945803485084]
  spiking neural networks (SNNs) are expected to realize artificial intelligence (AI) with less energy consumption. It provides a promising energy-efficient way for realistic control tasks by combining SNNs with deep reinforcement learning (RL)
  arXiv  Detail & Related papers  (2022-01-21T16:42:11Z)
• Single-Shot Pruning for Offline Reinforcement Learning [47.886329599997474]
  Deep Reinforcement Learning (RL) is a powerful framework for solving complex real-world problems. One way to tackle this problem is to prune neural networks leaving only the necessary parameters. We close the gap between RL and single-shot pruning techniques and present a general pruning approach to the Offline RL.
  arXiv  Detail & Related papers  (2021-12-31T18:10:02Z)
• RAPID-RL: A Reconfigurable Architecture with Preemptive-Exits for Efficient Deep-Reinforcement Learning [7.990007201671364]
  We propose a reconfigurable architecture with preemptive exits for efficient deep RL (RAPID-RL) RAPID-RL enables conditional activation of preemptive layers based on the difficulty level of inputs. We show that RAPID-RL incurs 0.34x (0.25x) number of operations (OPS) while maintaining performance above 0.88x (0.91x) on Atari (Drone navigation) tasks.
  arXiv  Detail & Related papers  (2021-09-16T21:30:40Z)
• Deep Reinforcement Learning with Population-Coded Spiking Neural Network for Continuous Control [0.0]
  We propose a population-coded spiking actor network (PopSAN) trained in conjunction with a deep critic network using deep reinforcement learning (DRL) We deployed the trained PopSAN on Intel's Loihi neuromorphic chip and benchmarked our method against the mainstream DRL algorithms for continuous control. Our results support the efficiency of neuromorphic controllers and suggest our hybrid RL as an alternative to deep learning, when both energy-efficiency and robustness are important.
  arXiv  Detail & Related papers  (2020-10-19T16:20:45Z)
• Optimizing Memory Placement using Evolutionary Graph Reinforcement Learning [56.83172249278467]
  We introduce Evolutionary Graph Reinforcement Learning (EGRL), a method designed for large search spaces. We train and validate our approach directly on the Intel NNP-I chip for inference. We additionally achieve 28-78% speed-up compared to the native NNP-I compiler on all three workloads.
  arXiv  Detail & Related papers  (2020-07-14T18:50:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.