Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers

• URL: http://arxiv.org/abs/2604.08336v1
• Date: Thu, 09 Apr 2026 15:09:02 GMT
• Title: Leveraging Complementary Embeddings for Replay Selection in Continual Learning with Small Buffers
• Abstract summary: Multiple Embedding Replay Selection, MERS, replaces the buffer selection module with a graph-based approach.<n>We show consistent improvements over SOTA selection strategies across a range of continual learning algorithms.<n>MERS is a practical, drop-in enhancement for replay-based continual learning.
• Score: 7.6344611819427035
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Catastrophic forgetting remains a key challenge in Continual Learning (CL). In replay-based CL with severe memory constraints, performance critically depends on the sample selection strategy for the replay buffer. Most existing approaches construct memory buffers using embeddings learned under supervised objectives. However, class-agnostic, self-supervised representations often encode rich, class-relevant semantics that are overlooked. We propose a new method, Multiple Embedding Replay Selection, MERS, which replaces the buffer selection module with a graph-based approach that integrates both supervised and self-supervised embeddings. Empirical results show consistent improvements over SOTA selection strategies across a range of continual learning algorithms, with particularly strong gains in low-memory regimes. On CIFAR-100 and TinyImageNet, MERS outperforms single-embedding baselines without adding model parameters or increasing replay volume, making it a practical, drop-in enhancement for replay-based continual learning.

Related papers

• Unsupervised Continual Learning with Growing Self-Organizing Maps and Synthetic Replay [47.020758735910306]
  This work presents a generative continual learning framework based on growing self-organizing maps (GSOMs) and encoder-decoder models for class-incremental learning.<n>The proposed approach enables exemplar-free replay using distributional statistical memory, which eliminates the need to store raw data.<n>Results across multiple benchmarks show that the proposed approach achieves performance competitive even with supervised state-of-the-art memory-based methods.
  arXiv  Detail & Related papers  (2026-08-27T19:48:44Z)
• MSSR: Memory-Aware Adaptive Replay for Continual LLM Fine-Tuning [36.119948355819]
  We propose an experience replay framework that estimates sample-level memory strength and schedules rehearsal at adaptive intervals to mitigate catastrophic forgetting.<n>Extensive experiments across three backbone models and 11 sequential tasks show that MSSR consistently outperforms state-of-the-art replay baselines.
  arXiv  Detail & Related papers  (2026-03-10T16:49:44Z)
• FOREVER: Forgetting Curve-Inspired Memory Replay for Language Model Continual Learning [63.20028888397869]
  FOREVER (FORgEtting curVe-inspired mEmory) is a novel framework that aligns replay schedules with a model-centric notion of time.<n>Building on this approach, FOREVER incorporates a forgetting curve-based replay scheduler to determine when to replay and an intensity-aware regularization mechanism to adaptively control how to replay.
  arXiv  Detail & Related papers  (2026-01-07T13:55:14Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Forget Forgetting: Continual Learning in a World of Abundant Memory [55.64184779530581]
  Continual learning has traditionally focused on minimizing exemplar memory.<n>This paper challenges this paradigm by investigating a more realistic regime.<n>We find that the core challenge shifts from stability to plasticity, as models become biased toward prior tasks and struggle to learn new ones.
  arXiv  Detail & Related papers  (2025-02-11T05:40:52Z)
• Watch Your Step: Optimal Retrieval for Continual Learning at Scale [1.7265013728931]
  In continual learning, a model learns incrementally over time while minimizing interference between old and new tasks. One of the most widely used approaches in continual learning is referred to as replay. We propose a framework for evaluating selective retrieval strategies, categorized by simple, independent class- and sample-selective primitives. We propose a set of strategies to prevent duplicate replays and explore whether new samples with low loss values can be learned without replay.
  arXiv  Detail & Related papers  (2024-04-16T17:35:35Z)
• Class-Wise Buffer Management for Incremental Object Detection: An Effective Buffer Training Strategy [11.109975137910881]
  Class incremental learning aims to solve a problem that arises when continuously adding unseen class instances to an existing model. We introduce an effective buffer training strategy (eBTS) that creates the optimized replay buffer on object detection.
  arXiv  Detail & Related papers  (2023-12-14T17:10:09Z)
• Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion [50.03041373044267]
  We propose a Supervised Contrastive learning framework with adaptive classification criterion for Continual Learning. Experiments show that CFL achieves state-of-the-art performance and has a stronger ability to overcome compared with the classification baselines.
  arXiv  Detail & Related papers  (2023-05-20T19:22:40Z)
• vCLIMB: A Novel Video Class Incremental Learning Benchmark [53.90485760679411]
  We introduce vCLIMB, a novel video continual learning benchmark. vCLIMB is a standardized test-bed to analyze catastrophic forgetting of deep models in video continual learning. We propose a temporal consistency regularization that can be applied on top of memory-based continual learning methods.
  arXiv  Detail & Related papers  (2022-01-23T22:14:17Z)
• GCR: Gradient Coreset Based Replay Buffer Selection For Continual Learning [1.911678487931003]
  We show significant gains (2%-4% absolute) over the state-of-the-art in the well-studied offline continual learning setting. Our findings also effectively transfer to online / streaming CL settings, showing upto 5% gains over existing approaches.
  arXiv  Detail & Related papers  (2021-11-18T18:01:57Z)
• An Investigation of Replay-based Approaches for Continual Learning [79.0660895390689]
  Continual learning (CL) is a major challenge of machine learning (ML) and describes the ability to learn several tasks sequentially without catastrophic forgetting (CF) Several solution classes have been proposed, of which so-called replay-based approaches seem very promising due to their simplicity and robustness. We empirically investigate replay-based approaches of continual learning and assess their potential for applications.
  arXiv  Detail & Related papers  (2021-08-15T15:05:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.