Balancing Plasticity and Stability with Fast and Slow Successor Features

• URL: http://arxiv.org/abs/2605.26357v2
• Date: Wed, 27 May 2026 14:46:11 GMT
• Title: Balancing Plasticity and Stability with Fast and Slow Successor Features
• Abstract summary: We study how stability and adaptation affect performance under continuous environmental change.<n>We find that methods favoring stability, such as synaptic consolidation, outperform approaches focused on plasticity.
• Score: 29.71873686385453
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A hallmark of intelligence is the ability to adapt in non-stationary environments, yet deep Reinforcement Learning (RL) agents often struggle in such settings. Prior studies introduce non-stationarity through abrupt shifts in features or dynamics, whereas real-world environments often evolve gradually through continual drift. This distinction has important implications for the "stability-plasticity dilemma" in RL, as abrupt task changes may demand more plasticity than naturalistic settings. To address this, we modify existing 3D Miniworld and MuJoCo environments to incorporate naturalistic, continual non-stationarity, and use them to examine how stability and adaptation affect performance under continuous environmental change. We find that methods favoring stability, such as synaptic consolidation, outperform approaches focused on plasticity, such as parameters resetting. Motivated by this result, and prior evidence that Successor Features (SFs) reduce interference, we investigate whether SFs are better consolidation targets than Q-values. Across both environments, applying neuro-inspired synaptic consolidation to SFs yields superior performance on continually changing settings. Moreover, consolidation is most effective when SFs are stabilized across multiple timescales, which capture complementary aspects of gradual environmental change. Together, these results suggest that stability is more critical in continual learning when changes are gradual, and that multi-timescale consolidation of predictive representations is an effective approach.

Related papers

• Dynamics Are Learned, Not Told: Semi-Supervised Discovery of Latent Dynamics Geometries For Zero-Shot Policy Adaptation [25.853116977213833]
  Real-world dynamics shifts pose a critical challenge for reinforcement learning in robotics.<n>Most existing methods rely on encoding explicitly identified physical parameters into a latent context.<n>We revisit dynamics adaptation from an outcome-centric perspective.
  arXiv  Detail & Related papers  (2026-06-01T14:05:23Z)
• When Sensors Fail: Temporal Sequence Models for Robust PPO under Sensor Drift [64.37959940809633]
  We study robustness of Proximal Policy Optimization (PPO) under temporally persistent sensor failures.<n>We show Transformer-based sequence policies substantially outperform, RNN, and SSMs in robustness, maintaining high returns even when large fractions of sensors are unavailable.
  arXiv  Detail & Related papers  (2026-03-04T22:21:54Z)
• On the Plasticity and Stability for Post-Training Large Language Models [54.757672540381236]
  We identify a root cause as the conflict between plasticity and stability gradients.<n>We propose Probabilistic Conflict Resolution (PCR), a framework that models gradients as random variables.<n>PCR significantly smooths the training trajectory and achieves superior performance in various reasoning tasks.
  arXiv  Detail & Related papers  (2026-02-06T07:31:26Z)
• Not All Preferences Are Created Equal: Stability-Aware and Gradient-Efficient Alignment for Reasoning Models [52.48582333951919]
  We propose a dynamic framework designed to enhance alignment reliability by maximizing the Signal-to-Noise Ratio of policy updates.<n>SAGE (Stability-Aware Gradient Efficiency) integrates a coarse-grained curriculum mechanism that refreshes candidate pools based on model competence.<n> Experiments on multiple mathematical reasoning benchmarks demonstrate that SAGE significantly accelerates convergence and outperforms static baselines.
  arXiv  Detail & Related papers  (2026-02-01T12:56:10Z)
• Soft Adaptive Policy Optimization [67.61886077470528]
  Reinforcement learning plays an increasingly important role in enhancing the reasoning capabilities of large language models.<n>Existing group-based policy optimization methods, such as GSPO and GRPO, alleviate this problem via hard clipping.<n>We propose Soft Adaptive Policy Optimization (SAPO), which replaces hard clipping with a smooth, temperature-controlled gate.
  arXiv  Detail & Related papers  (2025-11-25T14:25:19Z)
• Adapting to Fragmented and Evolving Data: A Fisher Information Perspective [0.0]
  FADE is a lightweight framework for robust learning under dynamic environments.<n>It employs a shift-aware regularization mechanism anchored in Fisher information geometry.<n>FADE operates online with fixed memory and no access to target labels.
  arXiv  Detail & Related papers  (2025-07-25T06:50:09Z)
• Noradrenergic-inspired gain modulation attenuates the stability gap in joint training [44.99833362998488]
  Studies in continual learning have identified a transient drop in performance on mastered tasks when assimilating new ones, known as the stability gap.<n>We argue that it reflects an imbalance between rapid adaptation and robust retention at task boundaries.<n>Inspired by locus coeruleus mediated noradrenergic bursts, we propose uncertainty-modulated gain dynamics.
  arXiv  Detail & Related papers  (2025-07-18T16:34:06Z)
• Maintain Plasticity in Long-timescale Continual Test-time Adaptation [25.795708177210876]
  Continual test-time domain adaptation aims to adjust pre-trained source models to perform well over time across non-stationary target environments.<n>The plasticity refers to the model's capability to adjust predictions in response to non-stationary environments continually.<n>We propose a simple yet effective policy, Adaptive Shrink-Restore (ASR), towards preserving the model's plasticity.
  arXiv  Detail & Related papers  (2024-12-28T05:47:40Z)
• Keep Moving: identifying task-relevant subspaces to maximise plasticity for newly learned tasks [0.22499166814992438]
  Continual learning algorithms strive to acquire new knowledge while preserving prior information. Often, these algorithms emphasise stability and restrict network updates upon learning new tasks. But is all change detrimental? We propose that activation spaces in neural networks can be decomposed into two subspaces.
  arXiv  Detail & Related papers  (2023-10-07T08:54:43Z)
• On the Sample Complexity and Metastability of Heavy-tailed Policy Search in Continuous Control [47.71156648737803]
  Reinforcement learning is a framework for interactive decision-making with incentives sequentially revealed across time without a system dynamics model. We characterize a defined defined chain, identifying that policies associated with Levy Processes of a tail index yield to wider peaks.
  arXiv  Detail & Related papers  (2021-06-15T20:12:44Z)
• The dynamical regime and its importance for evolvability, task performance and generalization [14.059479351946386]
  We find that all populations, regardless of their initial regime, evolve to be subcritical in simple tasks. We conclude that although the subcritical regime is preferable for a simple task, the optimal deviation from criticality depends on the task difficulty.
  arXiv  Detail & Related papers  (2021-03-22T21:22:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.