TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning

• URL: http://arxiv.org/abs/2604.25898v1
• Date: Tue, 28 Apr 2026 17:41:04 GMT
• Title: TSN-Affinity: Similarity-Driven Parameter Reuse for Continual Offline Reinforcement Learning
• Abstract summary: Continual offline reinforcement learning (CORL) aims to learn a sequence of tasks from datasets collected over time while preserving performance on previously learned tasks.<n>We propose TSN-Affinity, a novel CORL method based on TinySubNetworks and Decision Transformer.<n>We evaluate the approach on benchmarks based on Atari games and simulations of manipulation tasks with the Franka Emika Panda robotic arm.
• Score: 5.680044533158534
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual offline reinforcement learning (CORL) aims to learn a sequence of tasks from datasets collected over time while preserving performance on previously learned tasks. This setting corresponds to domains where new tasks arise over time, but adapting the model in live environment interactions is expensive, risky, or impossible. However, CORL inherits the dual difficulty of offline reinforcement learning and adapting while preventing catastrophic forgetting. Replay-based continual learning approaches remain a strong baseline but incur memory overhead and suffer from a distribution mismatch between replayed samples and newly learned policies. At the same time, architectural continual learning methods have shown strong potential in supervised learning but remain underexplored in CORL. In this work, we propose TSN-Affinity, a novel CORL method based on TinySubNetworks and Decision Transformer. The method enables task-specific parameterization and controlled knowledge sharing through a RL-aware reuse strategy that routes tasks according to action compatibility and latent similarity. We evaluate the approach on benchmarks based on Atari games and simulations of manipulation tasks with the Franka Emika Panda robotic arm, covering both discrete and continuous control. Results show strong retention from sparse SubNetworks, with routing further improving multi-task performance. Our findings suggest that similarity-guided architectural reuse is a strong and viable alternative to replay-based strategies in a CORL setting. Our code is available at: https://github.com/anonymized-for-submission123/tsn-affinity.

Related papers

• Trust Region Continual Learning as an Implicit Meta-Learner [3.705371747297478]
  We study a hybrid perspective: emphtrust region continual learning that combines generative replay with a Fisher-metric trust region constraint.<n>We show that, under local approximations, the resulting update admits a MAML-style interpretation with a single implicit inner step.<n>This yields an emergent meta-learning property in continual learning.
  arXiv  Detail & Related papers  (2026-02-02T18:19:16Z)
• HiCL: Hippocampal-Inspired Continual Learning [14.349317103576558]
  HiCL is a hippocampal-inspired dual-memory continual learning architecture.<n>We show how it can mitigate catastrophic forgetting by using elements inspired by the hippocampal circuitry.
  arXiv  Detail & Related papers  (2025-08-19T23:40:11Z)
• KAC: Kolmogorov-Arnold Classifier for Continual Learning [70.29494592027852]
  Continual learning requires models to train continuously across consecutive tasks without forgetting.<n>Most existing methods utilize linear classifiers, which struggle to maintain a stable classification space while learning new tasks.<n>Inspired by the success of Kolmogorov-Arnold Networks (KAN) in preserving learning during simple continual regression tasks, we set out to explore their potential in more complex continual learning scenarios.
  arXiv  Detail & Related papers  (2025-03-27T01:27:14Z)
• From Imitation to Refinement -- Residual RL for Precise Assembly [19.9786629249219]
  Recent advances in Behavior Cloning (BC) have made it easy to teach robots new tasks.<n>However, we find that the ease of teaching comes at the cost of unreliable performance.<n>We devise a simple yet effective method, ResiP, that overcomes the reliability problem while retaining BC's ease of teaching and long-horizon capabilities.
  arXiv  Detail & Related papers  (2024-07-23T17:44:54Z)
• Offline Reinforcement Learning from Datasets with Structured Non-Stationarity [50.35634234137108]
  Current Reinforcement Learning (RL) is often limited by the large amount of data needed to learn a successful policy. We address a novel Offline RL problem setting in which, while collecting the dataset, the transition and reward functions gradually change between episodes but stay constant within each episode. We propose a method based on Contrastive Predictive Coding that identifies this non-stationarity in the offline dataset, accounts for it when training a policy, and predicts it during evaluation.
  arXiv  Detail & Related papers  (2024-05-23T02:41:36Z)
• Solving Continual Offline Reinforcement Learning with Decision Transformer [78.59473797783673]
  Continuous offline reinforcement learning (CORL) combines continuous and offline reinforcement learning. Existing methods, employing Actor-Critic structures and experience replay (ER), suffer from distribution shifts, low efficiency, and weak knowledge-sharing. We introduce multi-head DT (MH-DT) and low-rank adaptation DT (LoRA-DT) to mitigate DT's forgetting problem.
  arXiv  Detail & Related papers  (2024-01-16T16:28:32Z)
• Action-Quantized Offline Reinforcement Learning for Robotic Skill Learning [68.16998247593209]
  offline reinforcement learning (RL) paradigm provides recipe to convert static behavior datasets into policies that can perform better than the policy that collected the data. In this paper, we propose an adaptive scheme for action quantization. We show that several state-of-the-art offline RL methods such as IQL, CQL, and BRAC improve in performance on benchmarks when combined with our proposed discretization scheme.
  arXiv  Detail & Related papers  (2023-10-18T06:07:10Z)
• OER: Offline Experience Replay for Continual Offline Reinforcement Learning [25.985985377992034]
  Continuously learning new skills via a sequence of pre-collected offline datasets is desired for an agent. In this paper, we formulate a new setting, continual offline reinforcement learning (CORL), where an agent learns a sequence of offline reinforcement learning tasks. We propose a new model-based experience selection scheme to build the replay buffer, where a transition model is learned to approximate the state distribution.
  arXiv  Detail & Related papers  (2023-05-23T08:16:44Z)
• Boosting Offline Reinforcement Learning via Data Rebalancing [104.3767045977716]
  offline reinforcement learning (RL) is challenged by the distributional shift between learning policies and datasets. We propose a simple yet effective method to boost offline RL algorithms based on the observation that resampling a dataset keeps the distribution support unchanged. We dub our method ReD (Return-based Data Rebalance), which can be implemented with less than 10 lines of code change and adds negligible running time.
  arXiv  Detail & Related papers  (2022-10-17T16:34:01Z)
• Learning Bayesian Sparse Networks with Full Experience Replay for Continual Learning [54.7584721943286]
  Continual Learning (CL) methods aim to enable machine learning models to learn new tasks without catastrophic forgetting of those that have been previously mastered. Existing CL approaches often keep a buffer of previously-seen samples, perform knowledge distillation, or use regularization techniques towards this goal. We propose to only activate and select sparse neurons for learning current and past tasks at any stage.
  arXiv  Detail & Related papers  (2022-02-21T13:25:03Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.