Robust Continual Unlearning against Knowledge Erosion and Forgetting Reversal

• URL: http://arxiv.org/abs/2604.19108v1
• Date: Tue, 21 Apr 2026 05:35:54 GMT
• Title: Robust Continual Unlearning against Knowledge Erosion and Forgetting Reversal
• Abstract summary: Machine unlearning plays a crucial role in realizing the right to be forgotten'' in artificial intelligence.<n>We evaluate existing unlearning algorithms in a more realistic scenario where unlearning is conducted repeatedly.<n>We propose SAFER, a continual unlearning framework that maintains representation stability for retain data while enforcing negative logit margins for forget data.
• Score: 31.09301411794716
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: As a means to balance the growth of the AI industry with the need for privacy protection, machine unlearning plays a crucial role in realizing the ``right to be forgotten'' in artificial intelligence. This technique enables AI systems to remove the influence of specific data while preserving the rest of the learned knowledge. Although it has been actively studied, most existing unlearning methods assume that unlearning is performed only once. In this work, we evaluate existing unlearning algorithms in a more realistic scenario where unlearning is conducted repeatedly, and in this setting, we identify two critical phenomena: (1) Knowledge Erosion, where the accuracy on retain data progressively degrades over unlearning phases, and (2) Forgetting Reversal, where previously forgotten samples become recognizable again in later phases. To address these challenges, we propose SAFER (StAbility-preserving Forgetting with Effective Regularization), a continual unlearning framework that maintains representation stability for retain data while enforcing negative logit margins for forget data. Extensive experiments show that SAFER mitigates not only knowledge erosion but also forgetting reversal, achieving stable performance across multiple unlearning phases.

Related papers

• Test-Time Unlearning via Sparse Autoencoder [55.4290003355764]
  We propose ARIA, a test-time unlearning method that leaves model weights intact and gates access to unwanted knowledge only when generation enters a forget-related state.<n>ARIA improves the forget-retain trade-off over weight-based baselines across both a thinking model and an instruction model.
  arXiv  Detail & Related papers  (2026-09-14T18:58:09Z)
• Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning [58.725080160369494]
  Distinguishable Deletion ($mathrmD2$) is a paradigm that restricts the response distribution in the latent representation rather than specific tokens.<n>We introduce an energy index that quantifies the presence of knowledge and the separation between unlearned and retained content.<n>Experiments demonstrate that EUA significantly outperforms previous methods, indicating the superiority of $mathrmD2$.
  arXiv  Detail & Related papers  (2026-05-16T03:15:35Z)
• SoK: Unlearnability and Unlearning for Model Dememorization [19.145486704492324]
  We present the first integrated analysis of model dememorization approaches leveraging unlearnability and unlearning.<n>Our contributions are threefold: (i) a unified taxonomy of unlearnability and scalable unlearning methods; (ii) an empirical evaluation revealing the robustness, interplay, and shallow dememorization of leading methods; and (iii) the first theoretical guarantee on dememorization depth for models processed through certified unlearning.
  arXiv  Detail & Related papers  (2026-05-12T06:17:13Z)
• Lethe:Adapter-Augmented Dual-Stream Update for Persistent Knowledge Erasure in Federated Unlearning [6.171968410497911]
  Federated unlearning (FU) aims to erase designated client-level, class-level, or sample-level knowledge from a global model.<n>We identify a critical failure mode, termed Knowledge resurfacing, by revealing that continued training can re-activate unlearned knowledge.<n>We propose Lethe, a novel federated unlearning method that de-correlates knowledge to be unlearned from knowledge to be retained.
  arXiv  Detail & Related papers  (2026-01-30T05:50:35Z)
• DUET: Distilled LLM Unlearning from an Efficiently Contextualized Teacher [5.406594712642111]
  Distilled Unlearning from an Efficient Teacher (DUET) is a novel distillation-based unlearning method.<n>It achieves higher performance in both forgetting and utility preservation, while being orders of magnitude more data-efficient than state-of-the-art unlearning methods.
  arXiv  Detail & Related papers  (2026-01-29T05:32:35Z)
• Understanding the Dilemma of Unlearning for Large Language Models [50.54260066313032]
  Unlearning seeks to remove specific knowledge from large language models (LLMs)<n>We propose unPact, an interpretable framework for unlearning via prompt attribution and contribution tracking.
  arXiv  Detail & Related papers  (2025-09-29T12:15:19Z)
• Erase or Hide? Suppressing Spurious Unlearning Neurons for Robust Unlearning [31.829376135133554]
  Large language models trained on web-scale data can memorize private or sensitive knowledge, raising significant privacy risks.<n>Some unlearning methods remain vulnerable to "relearning" during subsequent training, allowing a substantial portion of forgotten knowledge to resurface.<n>We introduce Ssiuu, a new class of unlearning methods that employs attribution-guided regularization to prevent spurious negative influence and faithfully remove target knowledge.
  arXiv  Detail & Related papers  (2025-09-26T12:27:23Z)
• Step-by-Step Reasoning Attack: Revealing 'Erased' Knowledge in Large Language Models [9.719371187651591]
  Unlearning techniques suppress and leave the knowledge beneath the surface, thus making it retrievable with the right prompts.<n>We introduce a step-by-step reasoning-based black-box attack, Sleek, that systematically exposes unlearning failures.<n>Of the generated adversarial prompts, 62.5% successfully retrieved forgotten Harry Potter facts from WHP-unlearned Llama, while 50% exposed unfair suppression of retained knowledge.
  arXiv  Detail & Related papers  (2025-06-14T04:22:17Z)
• UniErase: Towards Balanced and Precise Unlearning in Language Models [69.04923022755547]
  Large language models (LLMs) require iterative updates to address the outdated information problem.<n>UniErase is a novel unlearning framework that demonstrates precision and balanced performances between knowledge unlearning and ability retaining.
  arXiv  Detail & Related papers  (2025-05-21T15:53:28Z)
• Unlearning through Knowledge Overwriting: Reversible Federated Unlearning via Selective Sparse Adapter [35.65566527544619]
  Federated learning is a promising paradigm for privacy-preserving collaborative model training.<n>We propose FUSED, which first identifies critical layers by analyzing each layer's sensitivity to knowledge.<n> adapters are trained without altering the original parameters, overwriting the unlearning knowledge with the remaining knowledge.
  arXiv  Detail & Related papers  (2025-02-28T04:35:26Z)
• UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI [50.61495097098296]
  We revisit the paradigm in which unlearning is used for Large Language Models (LLMs) We introduce a concept of ununlearning, where unlearned knowledge gets reintroduced in-context. We argue that content filtering for impermissible knowledge will be required and even exact unlearning schemes are not enough for effective content regulation.
  arXiv  Detail & Related papers  (2024-06-27T10:24:35Z)
• Adaptively Integrated Knowledge Distillation and Prediction Uncertainty for Continual Learning [71.43841235954453]
  Current deep learning models often suffer from catastrophic forgetting of old knowledge when continually learning new knowledge. Existing strategies to alleviate this issue often fix the trade-off between keeping old knowledge (stability) and learning new knowledge (plasticity)
  arXiv  Detail & Related papers  (2023-01-18T05:36:06Z)
• Learning with Recoverable Forgetting [77.56338597012927]
  Learning wIth Recoverable Forgetting explicitly handles the task- or sample-specific knowledge removal and recovery. Specifically, LIRF brings in two innovative schemes, namely knowledge deposit and withdrawal. We conduct experiments on several datasets, and demonstrate that the proposed LIRF strategy yields encouraging results with gratifying generalization capability.
  arXiv  Detail & Related papers  (2022-07-17T16:42:31Z)
• Bilevel Continual Learning [76.50127663309604]
  We present a novel framework of continual learning named "Bilevel Continual Learning" (BCL) Our experiments on continual learning benchmarks demonstrate the efficacy of the proposed BCL compared to many state-of-the-art methods.
  arXiv  Detail & Related papers  (2020-07-30T16:00:23Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.