GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay

• URL: http://arxiv.org/abs/2508.04676v1
• Date: Wed, 06 Aug 2025 17:42:22 GMT
• Title: GeRe: Towards Efficient Anti-Forgetting in Continual Learning of LLM via General Samples Replay
• Abstract summary: We propose General Sample Replay (GeRe), a framework that use usual pretraining texts for efficient anti-forgetting.<n>We are the first to validate that a small, fixed set of pre-collected general replay samples is sufficient to resolve both concerns--retaining general capabilities while promoting overall performance.
• Score: 21.865853486614466
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The continual learning capability of large language models (LLMs) is crucial for advancing artificial general intelligence. However, continual fine-tuning LLMs across various domains often suffers from catastrophic forgetting, characterized by: 1) significant forgetting of their general capabilities, and 2) sharp performance declines in previously learned tasks. To simultaneously address both issues in a simple yet stable manner, we propose General Sample Replay (GeRe), a framework that use usual pretraining texts for efficient anti-forgetting. Beyond revisiting the most prevalent replay-based practices under GeRe, we further leverage neural states to introduce a enhanced activation states constrained optimization method using threshold-based margin (TM) loss, which maintains activation state consistency during replay learning. We are the first to validate that a small, fixed set of pre-collected general replay samples is sufficient to resolve both concerns--retaining general capabilities while promoting overall performance across sequential tasks. Indeed, the former can inherently facilitate the latter. Through controlled experiments, we systematically compare TM with different replay strategies under the GeRe framework, including vanilla label fitting, logit imitation via KL divergence and feature imitation via L1/L2 losses. Results demonstrate that TM consistently improves performance and exhibits better robustness. Our work paves the way for efficient replay of LLMs for the future. Our code and data are available at https://github.com/Qznan/GeRe.

Related papers

• Replay What Matters: Off-Policy Replay for Efficient LLM Reinforcement Unlearning [19.683904518981624]
  ReRULE is an off-policy replay enhancement for reinforcement unlearning.<n>We show that ReRULE yields a tighter hard-case convergence bound than pure on-policy RULE.
  arXiv  Detail & Related papers  (2026-06-13T14:52:40Z)
• Provable Effects of Data Replay in Continual Learning: A Feature Learning Perspective [28.881077229756404]
  We present a comprehensive theoretical framework for analyzing full data-replay training in continual learning.<n>We identify the signal-to-noise ratio (SNR) as a critical factor affecting forgetting.<n>We uncover a novel insight into task ordering: prioritizing higher-signal tasks not only facilitates learning of lower-signal tasks but also helps prevent catastrophic forgetting.
  arXiv  Detail & Related papers  (2026-02-02T20:21:17Z)
• Replay Failures as Successes: Sample-Efficient Reinforcement Learning for Instruction Following [42.05102776289243]
  Reinforcement Learning (RL) has shown promise for aligning Large Language Models (LLMs) to follow instructions with various constraints.<n>We propose Hindsight instruction Replay (HiR), a novel sample-efficient RL framework for complex instruction following tasks.
  arXiv  Detail & Related papers  (2025-12-29T13:31:08Z)
• Forget Less, Retain More: A Lightweight Regularizer for Rehearsal-Based Continual Learning [51.07663354001582]
  Deep neural networks suffer from catastrophic forgetting, where performance on previous tasks degrades after training on a new task.<n>We present a novel approach to address this challenge, focusing on the intersection of memory-based methods and regularization approaches.<n>We formulate a regularization strategy, termed Information Maximization (IM) regularizer, for memory-based continual learning methods.
  arXiv  Detail & Related papers  (2025-12-01T15:56:00Z)
• SuRe: Surprise-Driven Prioritised Replay for Continual LLM Learning [14.465381326192757]
  We argue that two failure modes drive this gap: selection (what to rehearse) and integration (how to consolidate new knowledge)<n>To address selection, we propose Surprise-prioritised Replay (SuRe), a simple, architecture-agnostic rule that ranks and stores the most surprising sequences.<n>To address integration, we add a dual-learner design with fast and slow LoRA adapters merged via an exponential moving average (EMA), enabling rapid adaptation while stabilising long-term knowledge.
  arXiv  Detail & Related papers  (2025-11-27T12:06:33Z)
• SFT Doesn't Always Hurt General Capabilities: Revisiting Domain-Specific Fine-Tuning in LLMs [53.77646961962239]
  Supervised Fine-Tuning (SFT) is a common approach to adapt Large Language Models (LLMs) to specialized tasks.<n>We show that SFT does not always hurt: using a smaller learning rate can substantially mitigate general performance degradation.
  arXiv  Detail & Related papers  (2025-09-25T05:28:22Z)
• Memorize or Generalize? Evaluating LLM Code Generation with Code Rewriting [54.48306552577881]
  We argue that large language models (LLMs) are mostly doing memorization (i.e., replicating or reusing large parts of their training data) versus generalization.<n>Existing evaluations largely proxy neglecting surface/structural similarity, thereby conflating benign reuse of repeated code with harmful recall and memorization task correctness.<n>We propose Memorization Risk Index (MRI), a normalized score that combines two signals: (i) how similar the model's answer for the rewritten task is to the original ground-truth solution, and (ii) how much performance drops from the original task to its rewritten counterpart.
  arXiv  Detail & Related papers  (2025-03-04T05:39:24Z)
• Experience Replay with Random Reshuffling [3.6622737533847936]
  In supervised learning, it is common to shuffle the dataset every epoch and consume data sequentially, which is called random reshuffling (RR)<n>We propose sampling methods that extend RR to experience replay, both in uniform and prioritized settings.<n>We evaluate our sampling methods on Atari benchmarks, demonstrating their effectiveness in deep reinforcement learning.
  arXiv  Detail & Related papers  (2025-03-04T04:37:22Z)
• Mitigating Forgetting in LLM Fine-Tuning via Low-Perplexity Token Learning [61.99353167168545]
  We show that fine-tuning with LLM-generated data improves target task performance and reduces non-target task degradation.<n>This is the first work to provide an empirical explanation based on token perplexity reduction to mitigate catastrophic forgetting in LLMs after fine-tuning.
  arXiv  Detail & Related papers  (2025-01-24T08:18:56Z)
• An Early FIRST Reproduction and Improvements to Single-Token Decoding for Fast Listwise Reranking [50.81324768683995]
  FIRST is a novel approach that integrates a learning-to-rank objective and leveraging the logits of only the first generated token. We extend the evaluation of FIRST to the TREC Deep Learning datasets (DL19-22), validating its robustness across diverse domains. Our experiments confirm that fast reranking with single-token logits does not compromise out-of-domain reranking quality.
  arXiv  Detail & Related papers  (2024-11-08T12:08:17Z)
• Investigating the Interplay of Prioritized Replay and Generalization [23.248982121562985]
  We study Prioritized Experience Replay (PER), where sampling is done proportionally to TD errors. PER is inspired by the success of prioritized sweeping in dynamic programming.
  arXiv  Detail & Related papers  (2024-07-12T21:56:24Z)
• FIRST: Faster Improved Listwise Reranking with Single Token Decoding [56.727761901751194]
  First, we introduce FIRST, a novel listwise LLM reranking approach leveraging the output logits of the first generated identifier to directly obtain a ranked ordering of the candidates. Empirical results demonstrate that FIRST accelerates inference by 50% while maintaining a robust ranking performance with gains across the BEIR benchmark. Our results show that LLM rerankers can provide a stronger distillation signal compared to cross-encoders, yielding substantial improvements in retriever recall after relevance feedback.
  arXiv  Detail & Related papers  (2024-06-21T21:27:50Z)
• Enhancing Consistency and Mitigating Bias: A Data Replay Approach for Incremental Learning [93.90047628101155]
  Deep learning systems are prone to catastrophic forgetting when learning from a sequence of tasks.<n>To address this, some methods propose replaying data from previous tasks during new task learning.<n>However, it is not expected in practice due to memory constraints and data privacy issues.
  arXiv  Detail & Related papers  (2024-01-12T12:51:12Z)
• Re-Reading Improves Reasoning in Large Language Models [87.46256176508376]
  We introduce a simple, yet general and effective prompting method, Re2, to enhance the reasoning capabilities of off-the-shelf Large Language Models (LLMs) Unlike most thought-eliciting prompting methods, such as Chain-of-Thought (CoT), Re2 shifts the focus to the input by processing questions twice, thereby enhancing the understanding process. We evaluate Re2 on extensive reasoning benchmarks across 14 datasets, spanning 112 experiments, to validate its effectiveness and generality.
  arXiv  Detail & Related papers  (2023-09-12T14:36:23Z)
• Mitigating the Learning Bias towards Repetition by Self-Contrastive Training for Open-Ended Generation [92.42032403795879]
  We show that pretrained language models (LMs) such as GPT2 still tend to generate repetitive texts. We attribute their overestimation of token-level repetition probabilities to the learning bias. We find that LMs use longer-range dependencies to predict repetitive tokens than non-repetitive ones, which may be the cause of sentence-level repetition loops.
  arXiv  Detail & Related papers  (2023-07-04T07:53:55Z)
• Generative Feature Replay with Orthogonal Weight Modification for Continual Learning [20.8966035274874]
  generative replay is a promising strategy which generates and replays pseudo data for previous tasks to alleviate catastrophic forgetting. We propose to replay penultimate layer feature with a generative model; 2) leverage a self-supervised auxiliary task to further enhance the stability of feature. Empirical results on several datasets show our method always achieves substantial improvement over powerful OWM.
  arXiv  Detail & Related papers  (2020-05-07T13:56:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.