Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning

• URL: http://arxiv.org/abs/2505.24061v1
• Date: Thu, 29 May 2025 23:07:58 GMT
• Title: Measure gradients, not activations! Enhancing neuronal activity in deep reinforcement learning
• Authors: Jiashun Liu, Zihao Wu, Johan Obando-Ceron, Pablo Samuel Castro, Aaron Courville, Ling Pan,
• Abstract summary: Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss.<n>GraMa (Gradient Magnitude Neural Activity Metric) is a metric for quantifying neuron-level learning capacity.<n>We show that GraMa effectively reveals persistent neuron inactivity across diverse architectures.
• Score: 25.277730616459255
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Deep reinforcement learning (RL) agents frequently suffer from neuronal activity loss, which impairs their ability to adapt to new data and learn continually. A common method to quantify and address this issue is the tau-dormant neuron ratio, which uses activation statistics to measure the expressive ability of neurons. While effective for simple MLP-based agents, this approach loses statistical power in more complex architectures. To address this, we argue that in advanced RL agents, maintaining a neuron's learning capacity, its ability to adapt via gradient updates, is more critical than preserving its expressive ability. Based on this insight, we shift the statistical objective from activations to gradients, and introduce GraMa (Gradient Magnitude Neural Activity Metric), a lightweight, architecture-agnostic metric for quantifying neuron-level learning capacity. We show that GraMa effectively reveals persistent neuron inactivity across diverse architectures, including residual networks, diffusion models, and agents with varied activation functions. Moreover, resetting neurons guided by GraMa (ReGraMa) consistently improves learning performance across multiple deep RL algorithms and benchmarks, such as MuJoCo and the DeepMind Control Suite.

Related papers

• Long-Tailed Data Classification by Increasing and Decreasing Neurons During Training [4.32776344138537]
  Real-world datasets often exhibit class imbalance situations where certain classes have far fewer samples than others.<n>We propose a method that periodically adds and removes neurons during training, thereby boosting representational power for minority classes.<n>Our results underscore the effectiveness of dynamic, biologically inspired network designs in improving performance on class-imbalanced data.
  arXiv  Detail & Related papers  (2025-07-14T05:29:16Z)
• Single-neuron deep generative model uncovers underlying physics of neuronal activity in Ca imaging data [0.0]
  We propose a novel framework for single-neuron representation learning using autoregressive variational autoencoders (AVAEs)<n>Our approach embeds individual neurons' signals into a reduced-dimensional space without the need for spike inference algorithms.<n>The AVAE excels over traditional linear methods by generating more informative and discriminative latent representations.
  arXiv  Detail & Related papers  (2025-01-24T16:33:52Z)
• Neuron Empirical Gradient: Discovering and Quantifying Neurons Global Linear Controllability [14.693407823048478]
  Our study first investigates the numerical relationship between neuron activations and model output.<n>We introduce NeurGrad, an accurate and efficient method for computing neuron empirical gradient (NEG)
  arXiv  Detail & Related papers  (2024-12-24T00:01:24Z)
• Growing Deep Neural Network Considering with Similarity between Neurons [4.32776344138537]
  We explore a novel approach of progressively increasing neuron numbers in compact models during training phases. We propose a method that reduces feature extraction biases and neuronal redundancy by introducing constraints based on neuron similarity distributions. Results on CIFAR-10 and CIFAR-100 datasets demonstrated accuracy improvement.
  arXiv  Detail & Related papers  (2024-08-23T11:16:37Z)
• Hadamard Representations: Augmenting Hyperbolic Tangents in RL [8.647614188619315]
  Activation functions are one of the key components of a deep neural network.<n>In reinforcement learning, the performance of continuously differentiable activations often falls short as compared to piece-wise linear functions.<n>We show that the dying neuron problem in RL is not exclusive to ReLUs and actually leads to additional problems in the case of continuously differentiable activations such as tanh.
  arXiv  Detail & Related papers  (2024-06-13T13:03:37Z)
• Hebbian Learning based Orthogonal Projection for Continual Learning of Spiking Neural Networks [74.3099028063756]
  We develop a new method with neuronal operations based on lateral connections and Hebbian learning. We show that Hebbian and anti-Hebbian learning on recurrent lateral connections can effectively extract the principal subspace of neural activities. Our method consistently solves for spiking neural networks with nearly zero forgetting.
  arXiv  Detail & Related papers  (2024-02-19T09:29:37Z)
• Modeling cognitive load as a self-supervised brain rate with electroencephalography and deep learning [2.741266294612776]
  This research presents a novel self-supervised method for mental workload modelling from EEG data. The method is a convolutional recurrent neural network trainable with spatially preserving spectral topographic head-maps from EEG data to fit the brain rate variable. Findings point to the existence of quasi-stable blocks of learnt high-level representations of cognitive activation because they can be induced through convolution and seem not to be dependent on each other over time, intuitively matching the non-stationary nature of brain responses.
  arXiv  Detail & Related papers  (2022-09-21T07:44:21Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)
• Reducing Catastrophic Forgetting in Self Organizing Maps with Internally-Induced Generative Replay [67.50637511633212]
  A lifelong learning agent is able to continually learn from potentially infinite streams of pattern sensory data. One major historic difficulty in building agents that adapt is that neural systems struggle to retain previously-acquired knowledge when learning from new samples. This problem is known as catastrophic forgetting (interference) and remains an unsolved problem in the domain of machine learning to this day.
  arXiv  Detail & Related papers  (2021-12-09T07:11:14Z)
• Dynamic Neural Diversification: Path to Computationally Sustainable Neural Networks [68.8204255655161]
  Small neural networks with a constrained number of trainable parameters, can be suitable resource-efficient candidates for many simple tasks. We explore the diversity of the neurons within the hidden layer during the learning process. We analyze how the diversity of the neurons affects predictions of the model.
  arXiv  Detail & Related papers  (2021-09-20T15:12:16Z)
• Towards Efficient Processing and Learning with Spikes: New Approaches for Multi-Spike Learning [59.249322621035056]
  We propose two new multi-spike learning rules which demonstrate better performance over other baselines on various tasks. In the feature detection task, we re-examine the ability of unsupervised STDP with its limitations being presented. Our proposed learning rules can reliably solve the task over a wide range of conditions without specific constraints being applied.
  arXiv  Detail & Related papers  (2020-05-02T06:41:20Z)
• Non-linear Neurons with Human-like Apical Dendrite Activations [81.18416067005538]
  We show that a standard neuron followed by our novel apical dendrite activation (ADA) can learn the XOR logical function with 100% accuracy. We conduct experiments on six benchmark data sets from computer vision, signal processing and natural language processing.
  arXiv  Detail & Related papers  (2020-02-02T21:09:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.