SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation

• URL: http://arxiv.org/abs/2310.12508v5
• Date: Thu, 4 Apr 2024 07:45:38 GMT
• Title: SalUn: Empowering Machine Unlearning via Gradient-based Weight Saliency in Both Image Classification and Generation
• Authors: Chongyu Fan, Jiancheng Liu, Yihua Zhang, Eric Wong, Dennis Wei, Sijia Liu,
• Abstract summary: We introduce the concept of 'weight saliency' for machine unlearning, drawing parallels with input saliency in model explanation. The resultant method that we call saliency unlearning (SalUn) narrows the performance gap with 'exact' unlearning. SalUn is the first principled MU approach that can effectively erase the influence of forgetting data, classes, or concepts in both image classification and generation tasks.
• Score: 30.168665935074166
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: With evolving data regulations, machine unlearning (MU) has become an important tool for fostering trust and safety in today's AI models. However, existing MU methods focusing on data and/or weight perspectives often suffer limitations in unlearning accuracy, stability, and cross-domain applicability. To address these challenges, we introduce the concept of 'weight saliency' for MU, drawing parallels with input saliency in model explanation. This innovation directs MU's attention toward specific model weights rather than the entire model, improving effectiveness and efficiency. The resultant method that we call saliency unlearning (SalUn) narrows the performance gap with 'exact' unlearning (model retraining from scratch after removing the forgetting data points). To the best of our knowledge, SalUn is the first principled MU approach that can effectively erase the influence of forgetting data, classes, or concepts in both image classification and generation tasks. As highlighted below, For example, SalUn yields a stability advantage in high-variance random data forgetting, e.g., with a 0.2% gap compared to exact unlearning on the CIFAR-10 dataset. Moreover, in preventing conditional diffusion models from generating harmful images, SalUn achieves nearly 100% unlearning accuracy, outperforming current state-of-the-art baselines like Erased Stable Diffusion and Forget-Me-Not. Codes are available at https://github.com/OPTML-Group/Unlearn-Saliency. (WARNING: This paper contains model outputs that may be offensive in nature.)

Related papers

• DESIRE: Dynamic Knowledge Consolidation for Rehearsal-Free Continual Learning [23.878495627964146]
  Continual learning aims to equip models with the ability to retain previously learned knowledge like a human. Existing methods usually overlook the issue of information leakage caused by the fact that the experiment data have been used in pre-trained models. In this paper, we propose a new LoRA-based rehearsal-free method named DESIRE.
  arXiv  Detail & Related papers  (2024-11-28T13:54:01Z)
• Attribute-to-Delete: Machine Unlearning via Datamodel Matching [65.13151619119782]
  Machine unlearning -- efficiently removing a small "forget set" training data on a pre-divertrained machine learning model -- has recently attracted interest. Recent research shows that machine unlearning techniques do not hold up in such a challenging setting.
  arXiv  Detail & Related papers  (2024-10-30T17:20:10Z)
• Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models [63.43422118066493]
  Machine unlearning (MU) is a crucial foundation for developing safe, secure, and trustworthy GenAI models. Traditional MU methods often rely on stringent assumptions and require access to real data. This paper introduces Score Forgetting Distillation (SFD), an innovative MU approach that promotes the forgetting of undesirable information in diffusion models.
  arXiv  Detail & Related papers  (2024-09-17T14:12:50Z)
• Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs [25.91643745340183]
  Large Language Models (LLMs) have demonstrated strong reasoning and memorization capabilities via pretraining on massive textual corpora. This poses risk of privacy and copyright violations, highlighting the need for efficient machine unlearning methods. We propose Low-rank Knowledge Unlearning (LoKU), a novel framework that enables robust and efficient unlearning for LLMs.
  arXiv  Detail & Related papers  (2024-08-13T04:18:32Z)
• Model Inversion Attacks Through Target-Specific Conditional Diffusion Models [54.69008212790426]
  Model inversion attacks (MIAs) aim to reconstruct private images from a target classifier's training set, thereby raising privacy concerns in AI applications. Previous GAN-based MIAs tend to suffer from inferior generative fidelity due to GAN's inherent flaws and biased optimization within latent space. We propose Diffusion-based Model Inversion (Diff-MI) attacks to alleviate these issues.
  arXiv  Detail & Related papers  (2024-07-16T06:38:49Z)
• Label Smoothing Improves Machine Unlearning [29.611981055071197]
  This work introduces UGradSL, a simple, plug-and-play machine unlearning approach that uses smoothed labels. The consistent improvement in MU performance is only at a marginal cost of additional computations.
  arXiv  Detail & Related papers  (2024-06-11T20:26:26Z)
• Challenging Forgets: Unveiling the Worst-Case Forget Sets in Machine Unlearning [9.998859702421417]
  Machine unlearning (MU) aims to eliminate the influence of chosen data points on model performance. Despite various MU methods for data influence erasure, evaluations have largely focused on random data forgetting. We propose identifying the data subset that presents the most significant challenge for influence erasure, pinpointing the worst-case forget set.
  arXiv  Detail & Related papers  (2024-03-12T06:50:32Z)
• Gradient Reweighting: Towards Imbalanced Class-Incremental Learning [8.438092346233054]
  Class-Incremental Learning (CIL) trains a model to continually recognize new classes from non-stationary data. A major challenge of CIL arises when applying to real-world data characterized by non-uniform distribution. We show that this dual imbalance issue causes skewed gradient updates with biased weights in FC layers, thus inducing over/under-fitting and catastrophic forgetting in CIL.
  arXiv  Detail & Related papers  (2024-02-28T18:08:03Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• Task-Aware Machine Unlearning and Its Application in Load Forecasting [4.00606516946677]
  This paper introduces the concept of machine unlearning which is specifically designed to remove the influence of part of the dataset on an already trained forecaster. A performance-aware algorithm is proposed by evaluating the sensitivity of local model parameter change using influence function and sample re-weighting. We tested the unlearning algorithms on linear, CNN, andMixer based load forecasters with a realistic load dataset.
  arXiv  Detail & Related papers  (2023-08-28T08:50:12Z)
• Model Sparsity Can Simplify Machine Unlearning [33.18951938708467]
  In response to recent data regulation requirements, machine unlearning (MU) has emerged as a critical process. Our study introduces a novel model-based perspective: model sparsification via weight pruning. We show in both theory and practice that model sparsity can boost the multi-criteria unlearning performance of an approximate unlearner.
  arXiv  Detail & Related papers  (2023-04-11T02:12:02Z)
• Non-Invasive Fairness in Learning through the Lens of Data Drift [88.37640805363317]
  We show how to improve the fairness of Machine Learning models without altering the data or the learning algorithm. We use a simple but key insight: the divergence of trends between different populations, and, consecutively, between a learned model and minority populations, is analogous to data drift. We explore two strategies (model-splitting and reweighing) to resolve this drift, aiming to improve the overall conformance of models to the underlying data.
  arXiv  Detail & Related papers  (2023-03-30T17:30:42Z)
• Self-Damaging Contrastive Learning [92.34124578823977]
  Unlabeled data in reality is commonly imbalanced and shows a long-tail distribution. This paper proposes a principled framework called Self-Damaging Contrastive Learning to automatically balance the representation learning without knowing the classes. Our experiments show that SDCLR significantly improves not only overall accuracies but also balancedness.
  arXiv  Detail & Related papers  (2021-06-06T00:04:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.