Unlearning through Knowledge Overwriting: Reversible Federated Unlearning via Selective Sparse Adapter

• URL: http://arxiv.org/abs/2502.20709v1
• Date: Fri, 28 Feb 2025 04:35:26 GMT
• Title: Unlearning through Knowledge Overwriting: Reversible Federated Unlearning via Selective Sparse Adapter
• Authors: Zhengyi Zhong, Weidong Bao, Ji Wang, Shuai Zhang, Jingxuan Zhou, Lingjuan Lyu, Wei Yang Bryan Lim,
• Abstract summary: Federated learning is a promising paradigm for privacy-preserving collaborative model training.<n>We propose FUSED, which first identifies critical layers by analyzing each layer's sensitivity to knowledge.<n> adapters are trained without altering the original parameters, overwriting the unlearning knowledge with the remaining knowledge.
• Score: 35.65566527544619
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Federated Learning is a promising paradigm for privacy-preserving collaborative model training. In practice, it is essential not only to continuously train the model to acquire new knowledge but also to guarantee old knowledge the right to be forgotten (i.e., federated unlearning), especially for privacy-sensitive information or harmful knowledge. However, current federated unlearning methods face several challenges, including indiscriminate unlearning of cross-client knowledge, irreversibility of unlearning, and significant unlearning costs. To this end, we propose a method named FUSED, which first identifies critical layers by analyzing each layer's sensitivity to knowledge and constructs sparse unlearning adapters for sensitive ones. Then, the adapters are trained without altering the original parameters, overwriting the unlearning knowledge with the remaining knowledge. This knowledge overwriting process enables FUSED to mitigate the effects of indiscriminate unlearning. Moreover, the introduction of independent adapters makes unlearning reversible and significantly reduces the unlearning costs. Finally, extensive experiments on three datasets across various unlearning scenarios demonstrate that FUSED's effectiveness is comparable to Retraining, surpassing all other baselines while greatly reducing unlearning costs.

Related papers

• FaithUn: Toward Faithful Forgetting in Language Models by Investigating the Interconnectedness of Knowledge [24.858928681280634]
  We define a new concept called superficial unlearning, which refers to the phenomenon where an unlearning method fails to erase interconnected knowledge.<n>Based on the definition, we introduce a new benchmark, FaithUn, to analyze and evaluate the faithfulness of unlearning in real-world knowledge QA settings.<n>We propose a novel unlearning method, KLUE, which updates only knowledge-related neurons to achieve faithful unlearning.
  arXiv  Detail & Related papers  (2025-02-26T15:11:03Z)
• Siamese Machine Unlearning with Knowledge Vaporization and Concentration [23.796344455232227]
  We propose the concepts of knowledge vaporization and concentration to selectively erase learned knowledge from specific data points. Our proposed Siamese unlearning method does not require additional memory overhead and full access to the remaining dataset.
  arXiv  Detail & Related papers  (2024-12-02T07:19:49Z)
• Learn while Unlearn: An Iterative Unlearning Framework for Generative Language Models [52.03511469562013]
  We introduce the Iterative Contrastive Unlearning (ICU) framework, which consists of three core components.<n>A Knowledge Unlearning Induction module targets specific knowledge for removal using an unlearning loss.<n>A Contrastive Learning Enhancement module preserves the model's expressive capabilities against the pure unlearning goal.<n>An Iterative Unlearning Refinement module dynamically adjusts the unlearning process through ongoing evaluation and updates.
  arXiv  Detail & Related papers  (2024-07-25T07:09:35Z)
• To Forget or Not? Towards Practical Knowledge Unlearning for Large Language Models [39.39428450239399]
  Large Language Models (LLMs) trained on extensive corpora inevitably retain sensitive data, such as personal privacy information and copyrighted material. Recent advancements in knowledge unlearning involve updating LLM parameters to erase specific knowledge. We introduce KnowUnDo to evaluate if the unlearning process inadvertently erases essential knowledge.
  arXiv  Detail & Related papers  (2024-07-02T03:34:16Z)
• UnUnlearning: Unlearning is not sufficient for content regulation in advanced generative AI [50.61495097098296]
  We revisit the paradigm in which unlearning is used for Large Language Models (LLMs) We introduce a concept of ununlearning, where unlearned knowledge gets reintroduced in-context. We argue that content filtering for impermissible knowledge will be required and even exact unlearning schemes are not enough for effective content regulation.
  arXiv  Detail & Related papers  (2024-06-27T10:24:35Z)
• Auto-selected Knowledge Adapters for Lifelong Person Re-identification [54.42307214981537]
  Lifelong Person Re-Identification requires systems to continually learn from non-overlapping datasets across different times and locations. Existing approaches, either rehearsal-free or rehearsal-based, still suffer from the problem of catastrophic forgetting. We introduce a novel framework AdalReID, that adopts knowledge adapters and a parameter-free auto-selection mechanism for lifelong learning.
  arXiv  Detail & Related papers  (2024-05-29T11:42:02Z)
• Class-wise Federated Unlearning: Harnessing Active Forgetting with Teacher-Student Memory Generation [11.638683787598817]
  We propose a neuro-inspired federated unlearning framework based on active forgetting. Our framework distinguishes itself from existing methods by utilizing new memories to overwrite old ones. Our method achieves satisfactory unlearning completeness against backdoor attacks.
  arXiv  Detail & Related papers  (2023-07-07T03:07:26Z)
• Anti-Retroactive Interference for Lifelong Learning [65.50683752919089]
  We design a paradigm for lifelong learning based on meta-learning and associative mechanism of the brain. It tackles the problem from two aspects: extracting knowledge and memorizing knowledge. It is theoretically analyzed that the proposed learning paradigm can make the models of different tasks converge to the same optimum.
  arXiv  Detail & Related papers  (2022-08-27T09:27:36Z)
• Learning with Recoverable Forgetting [77.56338597012927]
  Learning wIth Recoverable Forgetting explicitly handles the task- or sample-specific knowledge removal and recovery. Specifically, LIRF brings in two innovative schemes, namely knowledge deposit and withdrawal. We conduct experiments on several datasets, and demonstrate that the proposed LIRF strategy yields encouraging results with gratifying generalization capability.
  arXiv  Detail & Related papers  (2022-07-17T16:42:31Z)
• Transferability in Deep Learning: A Survey [80.67296873915176]
  The ability to acquire and reuse knowledge is known as transferability in deep learning. We present this survey to connect different isolated areas in deep learning with their relation to transferability. We implement a benchmark and an open-source library, enabling a fair evaluation of deep learning methods in terms of transferability.
  arXiv  Detail & Related papers  (2022-01-15T15:03:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.