The Lifecycle Principle: Stabilizing Dynamic Neural Networks with State Memory

• URL: http://arxiv.org/abs/2509.02575v2
• Date: Thu, 25 Sep 2025 10:54:37 GMT
• Title: The Lifecycle Principle: Stabilizing Dynamic Neural Networks with State Memory
• Abstract summary: I propose a regularization mechanism centered on state memory.<n>Instead of re-initializing a revived neuron, my method restores its parameters to their last known effective state.<n> Experiments on image classification benchmarks demonstrate that my method improves generalization and robustness.
• Score: 0.0
• License: http://creativecommons.org/licenses/by-sa/4.0/
• Abstract: I investigate a stronger form of regularization by deactivating neurons for extended periods, a departure from the temporary changes of methods like Dropout. However, this long-term dynamism introduces a critical challenge: severe training instability when neurons are revived with random weights. To solve this, I propose the Lifecycle (LC) principle, a regularization mechanism centered on a key innovation: state memory. Instead of re-initializing a revived neuron, my method restores its parameters to their last known effective state. This process preserves learned knowledge and avoids destructive optimization shocks. My theoretical analysis reveals that the LC principle smooths the loss landscape, guiding optimization towards flatter minima associated with better generalization. Experiments on image classification benchmarks demonstrate that my method improves generalization and robustness. Crucially, ablation studies confirm that state memory is essential for achieving these gains.

Related papers

• Calibrated Partial Resets: Preventing Policy Collapse in Continual Reinforcement Learning [3.1681974483553663]
  We propose Calibrated Partial Resets (CPR) to preserve plasticity without destabilizing training.<n>CPR avoids policy collapse over 400M training steps in SlipperyAnt.<n>CPR highlights utility-scaled reinitialization as a promising direction for continual learning.
  arXiv  Detail & Related papers  (2026-07-27T18:53:00Z)
• Relax Forcing: Relaxed KV-Memory for Consistent Long Video Generation [73.84423888025171]
  Autoregressive (AR) video diffusion has recently emerged as a promising paradigm for long video generation.<n>We show that extending generation to minute-scale horizons remains challenging due to progressive temporal degradation.<n>We introduce Relax Forcing, a structured temporal memory mechanism for AR diffusion.
  arXiv  Detail & Related papers  (2026-03-22T18:59:24Z)
• Stochastic Resetting Accelerates Policy Convergence in Reinforcement Learning [5.900714266080361]
  resetting is a powerful mechanism for optimizing first-passage properties.<n>We show that resetting accelerates convergence even when it does not reduce the search time of a purely diffusive agent.<n>In a continuous control task with neural-network-based value approximation, we show that random resetting improves deep reinforcement learning when exploration is difficult.
  arXiv  Detail & Related papers  (2026-03-17T17:50:32Z)
• Optimized Weight Initialization on the Stiefel Manifold for Deep ReLU Neural Networks [5.363441578662801]
  Improper weight training of ReLU networks can cause inactivation dying ReLU and exacerbate instability as network depth increases.<n>We introduce an optimization problem on the Stiefel manifold, thereby preserving scale and calibrating the pre-activation statistics.<n>We show that prevention of the dying ReLU problem, slower decay of activation variance, and mitigation of gradient vanishing, which together stabilize signal and gradient flow in deep architectures.
  arXiv  Detail & Related papers  (2025-08-30T05:17:31Z)
• AR-LIF: Adaptive reset leaky integrate-and-fire neuron for spiking neural networks [19.595600625488004]
  Spiking neural networks offer low energy consumption due to their event-driven nature.<n>We design an adaptive reset neuron that establishes relationships between inputs, outputs, and reset.<n> Experimental results demonstrate that our method achieves excellent performance while maintaining lower energy consumption.
  arXiv  Detail & Related papers  (2025-07-28T11:54:48Z)
• Forget Forgetting: Continual Learning in a World of Abundant Memory [55.64184779530581]
  Continual learning has traditionally focused on minimizing exemplar memory.<n>This paper challenges this paradigm by investigating a more realistic regime.<n>We find that the core challenge shifts from stability to plasticity, as models become biased toward prior tasks and struggle to learn new ones.
  arXiv  Detail & Related papers  (2025-02-11T05:40:52Z)
• Self-Normalized Resets for Plasticity in Continual Learning [6.5990719141691825]
  Self-Normalized Resets (SNR) is an adaptive algorithm that mitigates plasticity loss by resetting a neuron's weights when evidence suggests its firing rate has effectively dropped to zero.<n>We show that SNR consistently attains superior performance compared to its competitor algorithms.
  arXiv  Detail & Related papers  (2024-10-26T06:47:13Z)
• Fine-Grained Gradient Restriction: A Simple Approach for Mitigating Catastrophic Forgetting [41.891312602770746]
  Gradient Episodic Memory (GEM) achieves balance by utilizing a subset of past training samples to restrict the update direction of the model parameters. We show that memory strength is effective mainly because it improves GEM's ability generalization and therefore leads to a more favorable trade-off.
  arXiv  Detail & Related papers  (2024-10-01T17:03:56Z)
• On the Dynamics Under the Unhinged Loss and Beyond [104.49565602940699]
  We introduce the unhinged loss, a concise loss function, that offers more mathematical opportunities to analyze closed-form dynamics. The unhinged loss allows for considering more practical techniques, such as time-vary learning rates and feature normalization.
  arXiv  Detail & Related papers  (2023-12-13T02:11:07Z)
• FOSTER: Feature Boosting and Compression for Class-Incremental Learning [52.603520403933985]
  Deep neural networks suffer from catastrophic forgetting when learning new categories. We propose a novel two-stage learning paradigm FOSTER, empowering the model to learn new categories adaptively.
  arXiv  Detail & Related papers  (2022-04-10T11:38:33Z)
• Flattening Sharpness for Dynamic Gradient Projection Memory Benefits Continual Learning [67.99349091593324]
  We investigate the relationship between the weight loss landscape and sensitivity-stability in the continual learning scenario. Our proposed method consistently outperforms baselines with the superior ability to learn new skills while alleviating forgetting effectively.
  arXiv  Detail & Related papers  (2021-10-09T15:13:44Z)
• Training Feedback Spiking Neural Networks by Implicit Differentiation on the Equilibrium State [66.2457134675891]
  Spiking neural networks (SNNs) are brain-inspired models that enable energy-efficient implementation on neuromorphic hardware. Most existing methods imitate the backpropagation framework and feedforward architectures for artificial neural networks. We propose a novel training method that does not rely on the exact reverse of the forward computation.
  arXiv  Detail & Related papers  (2021-09-29T07:46:54Z)
• Gradient Descent on Neural Networks Typically Occurs at the Edge of Stability [94.4070247697549]
  Full-batch gradient descent on neural network training objectives operates in a regime we call the Edge of Stability. In this regime, the maximum eigenvalue of the training loss Hessian hovers just above the numerical value $2 / text(step size)$, and the training loss behaves non-monotonically over short timescales, yet consistently decreases over long timescales.
  arXiv  Detail & Related papers  (2021-02-26T22:08:19Z)
• The Golden Ratio of Learning and Momentum [0.5076419064097732]
  This paper proposes a new information-theoretical loss function motivated by neural signal processing in a synapse. All results taken together show that loss, learning rate, and momentum are closely connected.
  arXiv  Detail & Related papers  (2020-06-08T17:08:13Z)
• Revisiting Initialization of Neural Networks [72.24615341588846]
  We propose a rigorous estimation of the global curvature of weights across layers by approximating and controlling the norm of their Hessian matrix. Our experiments on Word2Vec and the MNIST/CIFAR image classification tasks confirm that tracking the Hessian norm is a useful diagnostic tool.
  arXiv  Detail & Related papers  (2020-04-20T18:12:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.