Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning

• URL: http://arxiv.org/abs/2510.12096v1
• Date: Tue, 14 Oct 2025 03:03:08 GMT
• Title: Rethinking the Role of Dynamic Sparse Training for Scalable Deep Reinforcement Learning
• Authors: Guozheng Ma, Lu Li, Zilin Wang, Haoyu Wang, Shengchao Hu, Leszek Rutkowski, Dacheng Tao,
• Abstract summary: Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL)<n>We show that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements.<n>We finally distill these insights into Module-Specific Training (MST), a practical framework that exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.
• Score: 58.533203990515034
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Scaling neural networks has driven breakthrough advances in machine learning, yet this paradigm fails in deep reinforcement learning (DRL), where larger models often degrade performance due to unique optimization pathologies such as plasticity loss. While recent works show that dynamically adapting network topology during training can mitigate these issues, existing studies have three critical limitations: (1) applying uniform dynamic training strategies across all modules despite encoder, critic, and actor following distinct learning paradigms, (2) focusing evaluation on basic architectures without clarifying the relative importance and interaction between dynamic training and architectural improvements, and (3) lacking systematic comparison between different dynamic approaches including sparse-to-sparse, dense-to-sparse, and sparse-to-dense. Through comprehensive investigation across modules and architectures, we reveal that dynamic sparse training strategies provide module-specific benefits that complement the primary scalability foundation established by architectural improvements. We finally distill these insights into Module-Specific Training (MST), a practical framework that further exploits the benefits of architectural improvements and demonstrates substantial scalability gains across diverse RL algorithms without algorithmic modifications.

Related papers

• Agentic-KGR: Co-evolutionary Knowledge Graph Construction through Multi-Agent Reinforcement Learning [6.665920297143511]
  Agentic-KGR is a novel framework enabling co-evolution between large language models (LLMs) and knowledge graphs (KGs)<n>Our approach introduces three key innovations: (1) a dynamic schema expansion mechanism that systematically extends graph beyond pre-defined boundaries during training; (2) a retrieval-augmented memory system enabling synergistic co-evolution between model parameters and knowledge structures through continuous optimization; and (3) a learnable multi-scale prompt compression approach that preserves critical information while reducing computational complexity through adaptive sequence optimization.
  arXiv  Detail & Related papers  (2025-10-10T09:00:07Z)
• Foundation Model for Skeleton-Based Human Action Understanding [56.89025287217221]
  This paper presents a Unified Skeleton-based Dense Representation Learning framework.<n>USDRL consists of a Transformer-based Dense Spatio-Temporal (DSTE), Multi-Grained Feature Decorrelation (MG-FD), and Multi-Perspective Consistency Training (MPCT)
  arXiv  Detail & Related papers  (2025-08-18T02:42:16Z)
• Stackelberg Coupling of Online Representation Learning and Reinforcement Learning [45.70357546589222]
  We introduce SCORER, a framework for value-based RL that views representation and Q-learning as two strategic agents in a hierarchical game.<n>Our proposed SCORER framework leads to a bi-level optimization problem whose solution is approximated by a two-timescale algorithm.
  arXiv  Detail & Related papers  (2025-08-10T18:36:54Z)
• Model Hemorrhage and the Robustness Limits of Large Language Models [119.46442117681147]
  Large language models (LLMs) demonstrate strong performance across natural language processing tasks, yet undergo significant performance degradation when modified for deployment.<n>We define this phenomenon as model hemorrhage - performance decline caused by parameter alterations and architectural changes.
  arXiv  Detail & Related papers  (2025-03-31T10:16:03Z)
• An Overview of Low-Rank Structures in the Training and Adaptation of Large Models [52.67110072923365]
  Recent research has uncovered a widespread phenomenon in deep networks: the emergence of low-rank structures.<n>These implicit low-dimensional patterns provide valuable insights for improving the efficiency of training and fine-tuning large-scale models.<n>We present a comprehensive review of advances in exploiting low-rank structures for deep learning and shed light on their mathematical foundations.
  arXiv  Detail & Related papers  (2025-03-25T17:26:09Z)
• Generalized Factor Neural Network Model for High-dimensional Regression [50.554377879576066]
  We tackle the challenges of modeling high-dimensional data sets with latent low-dimensional structures hidden within complex, non-linear, and noisy relationships.<n>Our approach enables a seamless integration of concepts from non-parametric regression, factor models, and neural networks for high-dimensional regression.
  arXiv  Detail & Related papers  (2025-02-16T23:13:55Z)
• Network Dynamics-Based Framework for Understanding Deep Neural Networks [8.376259475051974]
  We propose a theoretical framework to analyze learning dynamics through the lens of dynamical systems theory.<n>We redefine the notions of linearity and nonlinearity in neural networks by introducing two fundamental transformation units at the neuron level.<n>Different transformation modes lead to distinct collective behaviors in weight vector organization, different modes of information extraction, and the emergence of qualitatively different learning phases.
  arXiv  Detail & Related papers  (2025-01-05T04:23:21Z)
• Advancing Neural Network Performance through Emergence-Promoting Initialization Scheme [0.0]
  Emergence in machine learning refers to the spontaneous appearance of capabilities that arise from the scale and structure of training data.<n>We introduce a novel yet straightforward neural network initialization scheme that aims at achieving greater potential for emergence.<n>We demonstrate substantial improvements in both model accuracy and training speed, with and without batch normalization.
  arXiv  Detail & Related papers  (2024-07-26T18:56:47Z)
• Three Mechanisms of Feature Learning in a Linear Network [0.34530027457862006]
  We present an exact solution for the learning dynamics of a one-hidden-layer linear network, with one-dimensional data, across any finite width.<n>We identify three novel prototype mechanisms specific to the feature learning regime.<n>Our findings are substantiated with empirical evidence showing that these mechanisms also manifest in deep nonlinear networks handling real-world tasks.
  arXiv  Detail & Related papers  (2024-01-13T14:21:46Z)
• SIRe-Networks: Skip Connections over Interlaced Multi-Task Learning and Residual Connections for Structure Preserving Object Classification [28.02302915971059]
  In this paper, we introduce an interlaced multi-task learning strategy, defined SIRe, to reduce the vanishing gradient in relation to the object classification task. The presented methodology directly improves a convolutional neural network (CNN) by enforcing the input image structure preservation through auto-encoders. To validate the presented methodology, a simple CNN and various implementations of famous networks are extended via the SIRe strategy and extensively tested on the CIFAR100 dataset.
  arXiv  Detail & Related papers  (2021-10-06T13:54:49Z)
• Efficient Model-Based Multi-Agent Mean-Field Reinforcement Learning [89.31889875864599]
  We propose an efficient model-based reinforcement learning algorithm for learning in multi-agent systems. Our main theoretical contributions are the first general regret bounds for model-based reinforcement learning for MFC. We provide a practical parametrization of the core optimization problem.
  arXiv  Detail & Related papers  (2021-07-08T18:01:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.