Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning

• URL: http://arxiv.org/abs/2604.15414v1
• Date: Thu, 16 Apr 2026 17:06:54 GMT
• Title: Beyond Single-Model Optimization: Preserving Plasticity in Continual Reinforcement Learning
• Abstract summary: We introduce textscTeLAPA (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives.<n>In our MiniGrid CL setting, textscTeLAPA learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks.
• Score: 1.618563064839635
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual reinforcement learning must balance retention with adaptation, yet many methods still rely on \emph{single-model preservation}, committing to one evolving policy as the main reusable solution across tasks. Even when a previously successful policy is retained, it may no longer provide a reliable starting point for rapid adaptation after interference, reflecting a form of \emph{loss of plasticity} that single-policy preservation cannot address. Inspired by quality-diversity methods, we introduce \textsc{TeLAPA} (Transfer-Enabled Latent-Aligned Policy Archives), a continual RL framework that organizes behaviorally diverse policy neighborhoods into per-task archives and maintains a shared latent space so that archived policies remain comparable and reusable under non-stationary drift. This perspective shifts continual RL from retaining isolated solutions to maintaining \emph{skill-aligned neighborhoods} with competent and behaviorally related policies that support future relearning. In our MiniGrid CL setting, \textsc{TeLAPA} learns more tasks successfully, recovers competence faster on revisited tasks after interference, and retains higher performance across a sequence of tasks. Our analyses show that source-optimal policies are often not transfer-optimal, even within a local competent neighborhood, and that effective reuse depends on retaining and selecting among multiple nearby alternatives rather than collapsing them to one representative. Together, these results reframe continual RL around reusable and competent policy neighborhoods, providing a route beyond single-model preservation toward more plastic lifelong agents.

Related papers

• Multi-step Proximal Policy Improvement in Offline Reinforcement Learning [15.88392414749782]
  We develop a geometric view of offline actor updates by modeling policies as a probability manifold endowed with a chosen metric geometry.<n>We propose multi-step proximal policy improvement (MPI), a plug-in refinement mechanism that composes sequential re-centered proximal steps.<n>MPI enables controlled policy improvement beyond dataset support while retaining proximal control at each refinement.
  arXiv  Detail & Related papers  (2026-09-03T13:37:39Z)
• VIMPO: Value-Implicit Policy Optimization for LLMs [106.88933849641272]
  Group-relative methods such as GRPO avoid training a critic, but typically assign a trajectory-level advantage to every token.<n>Actor-critic methods provide denser learning signals, but require a learned value function with its own training instability.<n>We introduce VIMPO, a critic-free policy optimization method that derives a policy-implied value function from optimality conditions of KL-regularized reinforcement learning.
  arXiv  Detail & Related papers  (2026-06-18T09:44:12Z)
• Lever: Inference-Time Policy Reuse under Support Constraints [8.884389224195912]
  Reinforcement learning (RL) policies are typically trained for fixed objectives, making reuse difficult when task requirements change.<n>We study inference-time policy reuse: given a library of pre-trained policies and a new composite objective, can a high-quality policy be constructed entirely offline, without additional environment interaction?<n>We introduce lever, an end-to-end framework that retrieves relevant policies, evaluates them using behavioral embeddings, and composes new policies via offline Q-value composition.
  arXiv  Detail & Related papers  (2026-04-22T04:28:45Z)
• Latent Policy Steering through One-Step Flow Policies [34.06099184809882]
  offline reinforcement learning (RL) allows robots to learn from offline datasets without risky exploration.<n>Latent Policy Steering (LPS) enables high-fidelity latent policy improvement by backpropagating original-action-space Q-gradients through a differentiable one-step MeanFlow policy.<n>Across OGBench and real-world robotic tasks, LPS achieves state-of-the-art performance and consistently outperforms behavioral cloning and strong latent steering baselines.
  arXiv  Detail & Related papers  (2026-03-05T15:38:08Z)
• Preference Conditioned Multi-Objective Reinforcement Learning: Decomposed, Diversity-Driven Policy Optimization [2.595968385299781]
  Multi-objective reinforcement learning seeks to learn policies that balance multiple, often conflicting objectives.<n>We introduce $D3PO$, a PPO-based framework that reorganizes multi-objective policy optimization to address these issues directly.<n>$D3PO$ preserves per-objective learning signals through a decomposed optimization pipeline and integrates preferences only after stabilization.
  arXiv  Detail & Related papers  (2026-02-08T01:45:01Z)
• Learning Policy Representations for Steerable Behavior Synthesis [80.4542176039074]
  Given a Markov decision process (MDP), we seek to learn representations for a range of policies to facilitate behavior steering at test time.<n>We show that these representations can be approximated uniformly for a range of policies using a set-based architecture.<n>We use variational generative approach to induce a smooth latent space, and further shape it with contrastive learning so that latent distances align with differences in value functions.
  arXiv  Detail & Related papers  (2026-01-29T21:52:06Z)
• Polychromic Objectives for Reinforcement Learning [63.37185057794815]
  Reinforcement learning fine-tuning (RLFT) is a dominant paradigm for improving pretrained policies for downstream tasks.<n>We introduce an objective for policy methods that explicitly enforces the exploration and refinement of diverse generations.<n>We show how proximal policy optimization (PPO) can be adapted to optimize this objective.
  arXiv  Detail & Related papers  (2025-09-29T19:32:11Z)
• Continual Task Learning through Adaptive Policy Self-Composition [54.95680427960524]
  CompoFormer is a structure-based continual transformer model that adaptively composes previous policies via a meta-policy network. Our experiments reveal that CompoFormer outperforms conventional continual learning (CL) methods, particularly in longer task sequences.
  arXiv  Detail & Related papers  (2024-11-18T08:20:21Z)
• Agent based modelling for continuously varying supply chains [4.163948606359882]
  This paper seeks to address whether agents can control varying supply chain problems. Two state-of-the-art Reinforcement Learning (RL) algorithms are compared. Results show that more lean strategies adopted in Batch environments are different from those adopted in environments with varying products.
  arXiv  Detail & Related papers  (2023-12-24T15:04:46Z)
• IOB: Integrating Optimization Transfer and Behavior Transfer for Multi-Policy Reuse [50.90781542323258]
  Reinforcement learning (RL) agents can transfer knowledge from source policies to a related target task. Previous methods introduce additional components, such as hierarchical policies or estimations of source policies' value functions. We propose a novel transfer RL method that selects the source policy without training extra components.
  arXiv  Detail & Related papers  (2023-08-14T09:22:35Z)
• Optimistic Linear Support and Successor Features as a Basis for Optimal Policy Transfer [7.970144204429356]
  We introduce an SF-based extension of the Optimistic Linear Support algorithm to learn a set of policies whose SFs form a convex coverage set. We prove that policies in this set can be combined via generalized policy improvement to construct optimal behaviors for any new linearly-expressible tasks.
  arXiv  Detail & Related papers  (2022-06-22T19:00:08Z)
• Latent-Variable Advantage-Weighted Policy Optimization for Offline RL [70.01851346635637]
  offline reinforcement learning methods hold the promise of learning policies from pre-collected datasets without the need to query the environment for new transitions. In practice, offline datasets are often heterogeneous, i.e., collected in a variety of scenarios. We propose to leverage latent-variable policies that can represent a broader class of policy distributions. Our method improves the average performance of the next best-performing offline reinforcement learning methods by 49% on heterogeneous datasets.
  arXiv  Detail & Related papers  (2022-03-16T21:17:03Z)
• Offline Reinforcement Learning with Implicit Q-Learning [85.62618088890787]
  Current offline reinforcement learning methods need to query the value of unseen actions during training to improve the policy. We propose an offline RL method that never needs to evaluate actions outside of the dataset. This method enables the learned policy to improve substantially over the best behavior in the data through generalization.
  arXiv  Detail & Related papers  (2021-10-12T17:05:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.