SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning

• URL: http://arxiv.org/abs/2511.22367v1
• Date: Thu, 27 Nov 2025 12:06:33 GMT
• Title: SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning
• Abstract summary: We argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge)<n>To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising sequences.<n>To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge.
• Score: 14.465381326192757
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Continual learning, one's ability to adapt to a sequence of tasks without forgetting previously acquired knowledge, remains a major challenge in machine learning and a key gap between artificial and human intelligence. While regularisation and replay perform well in vision, they lag behind multi-task learning for large language models (LLMs), especially at scale with many tasks. We revisit replay and argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge). To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising (high Negative Log-Likelihood) sequences. SuRe achieves state-of-the-art performance in the Large Number of Tasks (LNT) setting and delivers the best overall average across both Standard CL and LNT benchmarks. To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge. Combining SuRe with the dual learner yields further gains, including improvements of up to +5 accuracy points on LNT over prior SOTA. Ablation studies confirm that our proposed method remains robust under reduced replay frequency and small buffer size, demonstrating both effectiveness and sample efficiency. Taken together, our results establish replay as a strong baseline for continual LLM fine-tuning and demonstrate that surprise-based selection and slow-weight consolidation are complementary components for mitigating catastrophic forgetting.

Related papers

• GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay [21.865853486614466]
  We propose General Sample Replay (GeRe), a framework that use usual pretraining texts for efficient anti-forgetting.<n>We are the first to validate that a small, fixed set of pre-collected general replay samples is sufficient to resolve both concerns--retaining general capabilities while promoting overall performance.
  arXiv  Detail & Related papers  (2025-08-06T17:42:22Z)
• Omni-Thinker: Scaling Multi-Task RL in LLMs with Hybrid Reward and Task Scheduling [66.0871543682453]
  We present Omni-Thinker, a unified reinforcement learning framework that scales large language models across diverse tasks.<n>Our scheduler orders tasks according to accuracy backward transfer (BWT), reducing forgetting and improving multi-task performance.
  arXiv  Detail & Related papers  (2025-07-20T01:50:16Z)
• Scaling Up RL: Unlocking Diverse Reasoning in LLMs via Prolonged Training [121.5858973157225]
  We investigate the effects of prolonged reinforcement learning on a small language model across a diverse set of reasoning domains.<n>We introduce controlled KL regularization, clipping ratio, and periodic reference policy resets as critical components for unlocking long-term performance gains.<n>Our model achieves significant improvements over strong baselines, including +14.7% on math, +13.9% on coding, and +54.8% on logic puzzle tasks.
  arXiv  Detail & Related papers  (2025-07-16T17:59:24Z)
• Reinforcement Learning Tuning for VideoLLMs: Reward Design and Data Efficiency [56.475612147721264]
  We propose a dual-reward formulation that supervises both semantic and temporal reasoning through discrete and continuous reward signals.<n>We evaluate our approach across eight representative video understanding tasks, including VideoQA, Temporal Video Grounding, and Grounded VideoQA.<n>Results underscore the importance of reward design and data selection in advancing reasoning-centric video understanding with MLLMs.
  arXiv  Detail & Related papers  (2025-06-02T17:28:26Z)
• Scalable Strategies for Continual Learning with Replay [0.0]
  We show that replay can play a foundational role in continual learning, allowing models to reconcile new information with past knowledge.<n>In practice, however, replay is quite unscalable, doubling the cost of continual learning when applied naively.<n>We introduce consolidation, a phasic approach to replay which leads to up to 55% less replay samples being needed for a given performance target.<n>Then, we propose sequential merging, an offshoot of task arithmetic which is tailored to the continual learning setting and is shown to work well in combination with replay.
  arXiv  Detail & Related papers  (2025-05-18T18:23:50Z)
• SD-LoRA: Scalable Decoupled Low-Rank Adaptation for Class Incremental Learning [73.93639228235622]
  Continual Learning with foundation models has emerged as a promising paradigm to exploit abundant knowledge acquired during pre-training for tackling sequential tasks.<n>Existing prompt-based and Low-Rank Adaptation-based (LoRA-based) methods often require expanding a prompt/LoRA pool or retaining samples of previous tasks.<n>We propose Scalable Decoupled LoRA (SD-LoRA) for class incremental learning, which continually separates the learning of the magnitude and direction of LoRA components without rehearsal.
  arXiv  Detail & Related papers  (2025-01-22T20:00:41Z)
• An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking [50.81324768683995]
  FIRST is a novel approach that integrates a learning-to-rank objective and leveraging the logits of only the first generated token. We extend the evaluation of FIRST to the TREC Deep Learning datasets (DL19-22), validating its robustness across diverse domains. Our experiments confirm that fast reranking with single-token logits does not compromise out-of-domain reranking quality.
  arXiv  Detail & Related papers  (2024-11-08T12:08:17Z)
• Replay-Free Continual Low-Rank Adaptation with Dynamic Memory [62.85596937435928]
  We revisit continual learning, which enables pre-trained vision transformers (ViTs) to sequentially fine-tune on new downstream tasks over time.<n>Recent studies highlight a crossover between CL techniques and parameter-efficient fine-tuning.<n>We propose a novel PEFT-CL method called Dual Low-Rank Adaptation (DualLoRA)
  arXiv  Detail & Related papers  (2024-11-01T14:28:39Z)
• LW2G: Learning Whether to Grow for Prompt-based Continual Learning [55.552510632228326]
  Recent Prompt-based Continual learning has achieved remarkable performance with pre-trained models.<n>These approaches expand a prompt pool by adding a new set of prompts while learning and select the correct set during inference.<n>Previous studies have revealed that learning task-wised prompt sets individually and low selection accuracy pose challenges to the performance of PCL.
  arXiv  Detail & Related papers  (2024-09-27T15:55:13Z)
• Generative Feature Replay with Orthogonal Weight Modification for Continual Learning [20.8966035274874]
  generative replay is a promising strategy which generates and replays pseudo data for previous tasks to alleviate catastrophic forgetting. We propose to replay penultimate layer feature with a generative model; 2) leverage a self-supervised auxiliary task to further enhance the stability of feature. Empirical results on several datasets show our method always achieves substantial improvement over powerful OWM.
  arXiv  Detail & Related papers  (2020-05-07T13:56:22Z)
• Continual Learning with Node-Importance based Adaptive Group Sparse Regularization [30.23319528662881]
  We propose a novel regularization-based continual learning method, dubbed as Adaptive Group Sparsity based Continual Learning (AGS-CL) Our method selectively employs the two penalties when learning each node based its the importance, which is adaptively updated after learning each new task.
  arXiv  Detail & Related papers  (2020-03-30T18:21:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.