Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning

• URL: http://arxiv.org/abs/2608.12874v1
• Date: Thu, 13 Aug 2026 06:38:04 GMT
• Title: Sustaining Plasticity via Learnable Wavelet Activations in Continual Learning
• Abstract summary: Plasticity loss has emerged as a critical challenge in continual learning.<n>We propose a novel learnable wavelet activation that decomposes the activation function into low-frequency and high-frequency components.
• Score: 23.89114000710872
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Plasticity loss has emerged as a critical challenge in continual learning that significantly hinders the acquisition of sequential tasks. While optimizing activation designs offers a potential solution, current fixed-form functions suffer from an inherent spectral bias towards low-frequency variations, whereas learnable variants permit unconstrained updates that induce catastrophic forgetting. To address these limitations, we propose a novel learnable wavelet activation that decomposes the activation function into low-frequency and high-frequency components to explicitly counter spectral bias. Furthermore, we employ dynamic wavelet injection to adaptively enhance plasticity for new tasks, alongside a regularization strategy to ensure the stability of previous learned knowledge. Theoretically, we provide rigorous mathematical guarantees for the proposed framework, proving the structural necessity of the hybrid wavelet architecture for efficient $L^2$ approximation and demonstrating that the decoupled learning rate mechanism successfully restores network plasticity for high-frequency information. Additionally, we provide a formal derivation of the loss-driven injection trigger mechanism to precisely guide the injection. Extensive empirical evaluations demonstrate that our approach maintains superior trainability and generalization throughout the learning process and achieves state-of-the-art performance across diverse continual learning benchmarks.

Related papers

• Energy-Structured Low-Rank Adaptation for Continual Learning [78.5390284258307]
  We propose textbfEnergy-Concentrated and textbfEnergy-Ordered textbfLow-textbfRank textbfAdaptation (E$2$-LoRA)<n>By explicitly ordering and concentrating knowledge into leading ranks, E$2$-LoRA frees capacity for subsequent tasks.
  arXiv  Detail & Related papers  (2026-05-26T12:55:02Z)
• A Systematic Post-Train Framework for Video Generation [76.26555417456773]
  Large-scale video diffusion models have demonstrated impressive capabilities in generating high-resolution and semantically rich content.<n>A significant gap remains between their pretraining performance and real-world deployment requirements due to critical issues such as prompt sensitivity, temporal inconsistency, and prohibitive inference costs.<n>We propose a comprehensive post-training framework that systematically aligns pretrained models with user intentions through four synergistic stages.
  arXiv  Detail & Related papers  (2026-04-28T09:34:51Z)
• Euphonium: Steering Video Flow Matching via Process Reward Gradient Guided Stochastic Dynamics [49.242224984144904]
  We propose Euphonium, a novel framework that steers generation via process reward gradient guided dynamics.<n>Our key insight is to formulate the sampling process as a theoretically principled algorithm that explicitly incorporates the gradient of a Process Reward Model.<n>We derive a distillation objective that internalizes the guidance signal into the flow network, eliminating inference-time dependency on the reward model.
  arXiv  Detail & Related papers  (2026-02-04T08:59:57Z)
• Autonomous Floquet Engineering of Bosonic Codes via Reinforcement Learning [4.78580425445851]
  Bosonic codes represent a promising route toward quantum error correction in continuous-variable computation systems.<n>We introduce a reinforcement-learning-assisted Floquet engineering approach for the autonomous preparation of bosonic codes that is general, efficient, and noise-resilient.
  arXiv  Detail & Related papers  (2025-10-25T09:14:55Z)
• Activation Function Design Sustains Plasticity in Continual Learning [1.618563064839635]
  In continual learning, models can progressively lose the ability to adapt.<n>We show that activation choice is a primary, architecture-agnostic lever for mitigating plasticity loss.
  arXiv  Detail & Related papers  (2025-09-26T16:41:47Z)
• Adaptive Variance-Penalized Continual Learning with Fisher Regularization [0.0]
  This work presents a novel continual learning framework that integrates Fisher-weighted asymmetric regularization of parameter variances.<n>Our method dynamically modulates regularization intensity according to parameter uncertainty, achieving enhanced stability and performance.
  arXiv  Detail & Related papers  (2025-08-15T21:49:28Z)
• Preserving Plasticity in Continual Learning with Adaptive Linearity Injection [10.641213440191551]
  Loss of plasticity in deep neural networks is the gradual reduction in a model's capacity to incrementally learn.<n>Recent work has shown that deep linear networks tend to be resilient towards loss of plasticity.<n>We propose Adaptive Linearization (AdaLin), a general approach that dynamically adapts each neuron's activation function to mitigate plasticity loss.
  arXiv  Detail & Related papers  (2025-05-14T15:36:51Z)
• Super Level Sets and Exponential Decay: A Synergistic Approach to Stable Neural Network Training [0.0]
  We develop a dynamic learning rate algorithm that integrates exponential decay and advanced anti-overfitting strategies. We prove that the superlevel sets of the loss function, as influenced by our adaptive learning rate, are always connected.
  arXiv  Detail & Related papers  (2024-09-25T09:27:17Z)
• Normalization and effective learning rates in reinforcement learning [52.59508428613934]
  Normalization layers have recently experienced a renaissance in the deep reinforcement learning and continual learning literature. We show that normalization brings with it a subtle but important side effect: an equivalence between growth in the norm of the network parameters and decay in the effective learning rate. We propose to make the learning rate schedule explicit with a simple re- parameterization which we call Normalize-and-Project.
  arXiv  Detail & Related papers  (2024-07-01T20:58:01Z)
• Provable Guarantees for Generative Behavior Cloning: Bridging Low-Level Stability and High-Level Behavior [51.60683890503293]
  We propose a theoretical framework for studying behavior cloning of complex expert demonstrations using generative modeling. We show that pure supervised cloning can generate trajectories matching the per-time step distribution of arbitrary expert trajectories.
  arXiv  Detail & Related papers  (2023-07-27T04:27:26Z)
• Imitating, Fast and Slow: Robust learning from demonstrations via decision-time planning [96.72185761508668]
  Planning at Test-time (IMPLANT) is a new meta-algorithm for imitation learning. We demonstrate that IMPLANT significantly outperforms benchmark imitation learning approaches on standard control environments.
  arXiv  Detail & Related papers  (2022-04-07T17:16:52Z)
• Reparameterized Variational Divergence Minimization for Stable Imitation [57.06909373038396]
  We study the extent to which variations in the choice of probabilistic divergence may yield more performant ILO algorithms. We contribute a re parameterization trick for adversarial imitation learning to alleviate the challenges of the promising $f$-divergence minimization framework. Empirically, we demonstrate that our design choices allow for ILO algorithms that outperform baseline approaches and more closely match expert performance in low-dimensional continuous-control tasks.
  arXiv  Detail & Related papers  (2020-06-18T19:04:09Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.