The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning

• URL: http://arxiv.org/abs/2606.29832v1
• Date: Mon, 29 Jun 2026 06:18:51 GMT
• Title: The Forgetting-Retention Dilemma: Certified Unlearning Theory in Continual Learning
• Abstract summary: Machine unlearning aims to eliminate influence of specific data from models to safeguard privacy.<n>Current certified unlearning algorithms fail to account for cumulative model evolution inherent to continuous learning.<n>We formulate CL's unlearning objective as the minimization of post-unlearning excess risk, which decomposes into CL excess risk and unlearning loss.
• Score: 23.209639058960477
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning aims to eliminate the influence of specific data from trained models to safeguard privacy. However, this presents a significant challenge in the context of continual learning (CL), where models update sequentially on dynamic datasets. A major limitation is that current certified unlearning algorithms fail to account for the complex, cumulative model evolution inherent to CL framework. In this work, we establish the first theoretical foundation bridging CL and machine unlearning. We formulate the CL's unlearning objective as the minimization of post-unlearning excess risk, which decomposes into CL excess risk and unlearning loss, characterizing the fundamental trade-off between preserving historical knowledge and targeted forgetting. Under mild assumptions, we first establish an upper bound for the CL excess risk in non-convex models. We then adapt two certified unlearning approaches, gradient-based and Hessian-based, to the CL framework. Our analysis reveals that while the gradient-based approach is less effective than the Hessian-based method in minimizing unlearning loss, it offers the distinct advantage of nearly zero storage overhead for enabling unlearning. This insight motivates a hybrid strategy that reduces storage costs while maintaining post-unlearning performance. Experimental results further validate our theoretical findings.

Related papers

• To Retain or to Adapt? Generalizing Continual Learning [39.95178050304739]
  We argue that in non-stationary environments prioritizing retention can impede real-time adaptation.<n>We formalize CL as an online optimization problem governed by the interaction between environmental and learning dynamics.<n>We propose a new general class of continual learning algorithms, which we call Predictive Continual Learning.
  arXiv  Detail & Related papers  (2026-07-06T20:11:53Z)
• IDER: IDempotent Experience Replay for Reliable Continual Learning [36.35747221992653]
  Catastrophic forgetting is a major challenge in continual learning (CL)<n>Existing uncertainty-aware CL methods suffer from high computational overhead and incompatibility with mainstream replay methods.<n>We propose idempotent experience replay (IDER), a novel approach based on the idempotent property where repeated function applications yield the same output.
  arXiv  Detail & Related papers  (2026-02-28T12:34:18Z)
• A Unified Gradient-based Framework for Task-agnostic Continual Learning-Unlearning [30.2773429357068]
  Recent advancements in deep models have highlighted the need for intelligent systems that combine continual learning (CL) for knowledge acquisition with machine unlearning (MU) for data removal.<n>We reveal their intrinsic connection through a unified optimization framework based on Kullback-Leibler divergence minimization.<n>Experiments demonstrate that the proposed UG-CLU framework effectively coordinates incremental learning, precise unlearning, and knowledge stability across multiple datasets and model architectures.
  arXiv  Detail & Related papers  (2025-05-21T06:49:05Z)
• Adaptive Rank, Reduced Forgetting: Knowledge Retention in Continual Learning Vision-Language Models with Dynamic Rank-Selective LoRA [26.079123341965687]
  We study low-rank learning and analyze how LoRA ranks and placements affect learning and forgetting.<n>A higher-rank LoRA improves task learning (plasticity) but increases forgetting, while a lower-rank LoRA enhances stability but limits adaptation.<n>Motivated by this, we propose Continual Dynamic Rank-Selective LoRA (CoDyRA), which continually updates PTMs with LoRA adapters of adaptively optimized ranks.
  arXiv  Detail & Related papers  (2024-12-01T23:41:42Z)
• Temporal-Difference Variational Continual Learning [77.92320830700797]
  We propose new learning objectives that integrate the regularization effects of multiple previous posterior estimations.<n>Our approach effectively mitigates Catastrophic Forgetting, outperforming strong Variational CL methods.
  arXiv  Detail & Related papers  (2024-10-10T10:58:41Z)
• LoRanPAC: Low-rank Random Features and Pre-trained Models for Bridging Theory and Practice in Continual Learning [103.45785408116146]
  Continual learning (CL) aims to train a model that can solve multiple tasks presented sequentially.<n>Recent CL approaches have achieved strong performance by leveraging large pre-trained models that generalize well to downstream tasks.<n>However, such methods lack theoretical guarantees, making them prone to unexpected failures.<n>We aim to bridge this gap by designing a simple CL method that is theoretically sound and highly performant.
  arXiv  Detail & Related papers  (2024-10-01T12:58:37Z)
• Enhancing Robustness of Vision-Language Models through Orthogonality Learning and Self-Regularization [77.62516752323207]
  We introduce an orthogonal fine-tuning method for efficiently fine-tuning pretrained weights and enabling enhanced robustness and generalization. A self-regularization strategy is further exploited to maintain the stability in terms of zero-shot generalization of VLMs, dubbed OrthSR. For the first time, we revisit the CLIP and CoOp with our method to effectively improve the model on few-shot image classficiation scenario.
  arXiv  Detail & Related papers  (2024-07-11T10:35:53Z)
• What Makes CLIP More Robust to Long-Tailed Pre-Training Data? A Controlled Study for Transferable Insights [67.72413262980272]
  Severe data imbalance naturally exists among web-scale vision-language datasets. We find CLIP pre-trained thereupon exhibits notable robustness to the data imbalance compared to supervised learning. The robustness and discriminability of CLIP improve with more descriptive language supervision, larger data scale, and broader open-world concepts.
  arXiv  Detail & Related papers  (2024-05-31T17:57:24Z)
• Data Poisoning for In-context Learning [49.77204165250528]
  In-context learning (ICL) has been recognized for its innovative ability to adapt to new tasks.<n>This paper delves into the critical issue of ICL's susceptibility to data poisoning attacks.<n>We introduce ICLPoison, a specialized attacking framework conceived to exploit the learning mechanisms of ICL.
  arXiv  Detail & Related papers  (2024-02-03T14:20:20Z)
• Few-Shot Class-Incremental Learning with Prior Knowledge [94.95569068211195]
  We propose Learning with Prior Knowledge (LwPK) to enhance the generalization ability of the pre-trained model. Experimental results indicate that LwPK effectively enhances the model resilience against catastrophic forgetting.
  arXiv  Detail & Related papers  (2024-02-02T08:05:35Z)
• Mitigating Forgetting in Online Continual Learning via Contrasting Semantically Distinct Augmentations [22.289830907729705]
  Online continual learning (OCL) aims to enable model learning from a non-stationary data stream to continuously acquire new knowledge as well as retain the learnt one. Main challenge comes from the "catastrophic forgetting" issue -- the inability to well remember the learnt knowledge while learning the new ones.
  arXiv  Detail & Related papers  (2022-11-10T05:29:43Z)
• Online Continual Learning with Contrastive Vision Transformer [67.72251876181497]
  This paper proposes a framework Contrastive Vision Transformer (CVT) to achieve a better stability-plasticity trade-off for online CL. Specifically, we design a new external attention mechanism for online CL that implicitly captures previous tasks' information. Based on the learnable focuses, we design a focal contrastive loss to rebalance contrastive learning between new and past classes and consolidate previously learned representations.
  arXiv  Detail & Related papers  (2022-07-24T08:51:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.