An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms

• URL: http://arxiv.org/abs/2509.23750v1
• Date: Sun, 28 Sep 2025 08:54:33 GMT
• Title: An Investigation of Batch Normalization in Off-Policy Actor-Critic Algorithms
• Authors: Li Wang, Sudun, Xingjian Zhang, Wenjun Wu, Lei Huang,
• Abstract summary: Batch Normalization (BN) has played a pivotal role in the success of deep learning by improving training stability, mitigating overfitting, and enabling more effective optimization.<n>We argue that BN retains unique advantages in deep reinforcement learning settings, particularly through its robustness and its ability to ease training.<n>We propose the Mode-Aware Batch Normalization (MA-BN) method with practical actionable recommendations for robust BN integration in DRL pipelines.
• Score: 9.999241269705744
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Batch Normalization (BN) has played a pivotal role in the success of deep learning by improving training stability, mitigating overfitting, and enabling more effective optimization. However, its adoption in deep reinforcement learning (DRL) has been limited due to the inherent non-i.i.d. nature of data and the dynamically shifting distributions induced by the agent's learning process. In this paper, we argue that, despite these challenges, BN retains unique advantages in DRL settings, particularly through its stochasticity and its ability to ease training. When applied appropriately, BN can adapt to evolving data distributions and enhance both convergence speed and final performance. To this end, we conduct a comprehensive empirical study on the use of BN in off-policy actor-critic algorithms, systematically analyzing how different training and evaluation modes impact performance. We further identify failure modes that lead to instability or divergence, analyze their underlying causes, and propose the Mode-Aware Batch Normalization (MA-BN) method with practical actionable recommendations for robust BN integration in DRL pipelines. We also empirically validate that, in RL settings, MA-BN accelerates and stabilizes training, broadens the effective learning rate range, enhances exploration, and reduces overall optimization difficulty. Our code is available at: https://github.com/monster476/ma-bn.git.

Related papers

• Towards Stable and Effective Reinforcement Learning for Mixture-of-Experts [113.0656076371565]
  We propose a novel router-aware approach to optimize importance sampling weights in off-policy reinforcement learning (RL)<n>Specifically, we design a rescaling strategy guided by router logits, which effectively reduces gradient variance and mitigates training divergence.<n> Experimental results demonstrate that our method significantly improves both the convergence stability and the final performance of MoE models.
  arXiv  Detail & Related papers  (2025-10-27T05:47:48Z)
• CurES: From Gradient Analysis to Efficient Curriculum Learning for Reasoning LLMs [53.749193998004166]
  Curriculum learning plays a crucial role in enhancing the training efficiency of large language models.<n>We propose CurES, an efficient training method that accelerates convergence and employs Bayesian posterior estimation to minimize computational overhead.
  arXiv  Detail & Related papers  (2025-10-01T15:41:27Z)
• CaRe-BN: Precise Moving Statistics for Stabilizing Spiking Neural Networks in Reinforcement Learning [50.87795054453648]
  Spiking Neural Networks (SNNs) offer low-latency and energy-efficient decision-making on neuromorphic hardware.<n>Due to the discrete and non-differentiable nature of spikes, directly trained SNNs rely heavily on Batch Normalization (BN) to stabilize gradient updates.<n>In online Reinforcement Learning (RL), BN statistics hinder exploitation, resulting in slower convergence and suboptimal policies.
  arXiv  Detail & Related papers  (2025-09-28T10:21:17Z)
• Improving Multi-Step Reasoning Abilities of Large Language Models with Direct Advantage Policy Optimization [22.67700436936984]
  We introduce Direct Advantage Policy Optimization (DAPO), a novel step-level offline reinforcement learning algorithm.<n>DAPO employs a critic function to predict the reasoning accuracy at each step, thereby generating dense signals to refine the generation strategy.<n>Our results show that DAPO can effectively enhance the mathematical and code capabilities on both SFT models and RL models, demonstrating the effectiveness of DAPO.
  arXiv  Detail & Related papers  (2024-12-24T08:39:35Z)
• Unified Batch Normalization: Identifying and Alleviating the Feature Condensation in Batch Normalization and a Unified Framework [55.22949690864962]
  Batch Normalization (BN) has become an essential technique in contemporary neural network design. We propose a two-stage unified framework called Unified Batch Normalization (UBN) UBN significantly enhances performance across different visual backbones and different vision tasks.
  arXiv  Detail & Related papers  (2023-11-27T16:41:31Z)
• Hybrid Reinforcement Learning for Optimizing Pump Sustainability in Real-World Water Distribution Networks [55.591662978280894]
  This article addresses the pump-scheduling optimization problem to enhance real-time control of real-world water distribution networks (WDNs) Our primary objectives are to adhere to physical operational constraints while reducing energy consumption and operational costs. Traditional optimization techniques, such as evolution-based and genetic algorithms, often fall short due to their lack of convergence guarantees.
  arXiv  Detail & Related papers  (2023-10-13T21:26:16Z)
• Overcoming Recency Bias of Normalization Statistics in Continual Learning: Balance and Adaptation [67.77048565738728]
  Continual learning involves learning a sequence of tasks and balancing their knowledge appropriately. We propose Adaptive Balance of BN (AdaB$2$N), which incorporates appropriately a Bayesian-based strategy to adapt task-wise contributions. Our approach achieves significant performance gains across a wide range of benchmarks.
  arXiv  Detail & Related papers  (2023-10-13T04:50:40Z)
• Efficient Deep Reinforcement Learning Requires Regulating Overfitting [91.88004732618381]
  We show that high temporal-difference (TD) error on the validation set of transitions is the main culprit that severely affects the performance of deep RL algorithms. We show that a simple online model selection method that targets the validation TD error is effective across state-based DMC and Gym tasks.
  arXiv  Detail & Related papers  (2023-04-20T17:11:05Z)
• Learning to Optimize for Reinforcement Learning [58.01132862590378]
  Reinforcement learning (RL) is essentially different from supervised learning, and in practice, these learneds do not work well even in simple RL tasks. Agent-gradient distribution is non-independent and identically distributed, leading to inefficient meta-training. We show that, although only trained in toy tasks, our learned can generalize unseen complex tasks in Brax.
  arXiv  Detail & Related papers  (2023-02-03T00:11:02Z)
• Counterbalancing Teacher: Regularizing Batch Normalized Models for Robustness [15.395021925719817]
  Batch normalization (BN) is a technique for training deep neural networks that accelerates their convergence to reach higher accuracy. We show that BN incentivizes the model to rely on low-variance features that are highly specific to the training (in-domain) data. We propose Counterbalancing Teacher (CT) to enforce the student network's learning of robust representations.
  arXiv  Detail & Related papers  (2022-07-04T16:16:24Z)
• Continual Normalization: Rethinking Batch Normalization for Online Continual Learning [21.607816915609128]
  We study the cross-task normalization effect of Batch Normalization (BN) in online continual learning. BN normalizes the testing data using moments biased towards the current task, resulting in higher catastrophic forgetting. We propose Continual Normalization (CN) to facilitate training similar to BN while mitigating its negative effect.
  arXiv  Detail & Related papers  (2022-03-30T07:23:24Z)
• Rebalancing Batch Normalization for Exemplar-based Class-Incremental Learning [23.621259845287824]
  Batch Normalization (BN) has been extensively studied for neural nets in various computer vision tasks. We develop a new update patch for BN, particularly tailored for the exemplar-based class-incremental learning (CIL)
  arXiv  Detail & Related papers  (2022-01-29T11:03:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.