Deep Unlearning: Fast and Efficient Gradient-free Approach to Class Forgetting

• URL: http://arxiv.org/abs/2312.00761v4
• Date: Mon, 5 Aug 2024 18:40:07 GMT
• Title: Deep Unlearning: Fast and Efficient Gradient-free Approach to Class Forgetting
• Authors: Sangamesh Kodge, Gobinda Saha, Kaushik Roy,
• Abstract summary: We introduce a novel class unlearning algorithm designed to strategically eliminate specific classes from the learned model. Our algorithm exhibits competitive unlearning performance and resilience against Membership Inference Attacks (MIA)
• Score: 9.91998873101083
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning is a prominent and challenging field, driven by regulatory demands for user data deletion and heightened privacy awareness. Existing approaches involve retraining model or multiple finetuning steps for each deletion request, often constrained by computational limits and restricted data access. In this work, we introduce a novel class unlearning algorithm designed to strategically eliminate specific classes from the learned model. Our algorithm first estimates the Retain and the Forget Spaces using Singular Value Decomposition on the layerwise activations for a small subset of samples from the retain and unlearn classes, respectively. We then compute the shared information between these spaces and remove it from the forget space to isolate class-discriminatory feature space. Finally, we obtain the unlearned model by updating the weights to suppress the class discriminatory features from the activation spaces. We demonstrate our algorithm's efficacy on ImageNet using a Vision Transformer with only $\sim 1.5\%$ drop in retain accuracy compared to the original model while maintaining under $1\%$ accuracy on the unlearned class samples. Furthermore, our algorithm exhibits competitive unlearning performance and resilience against Membership Inference Attacks (MIA). Compared to baselines, it achieves an average accuracy improvement of $1.38\%$ on the ImageNet dataset while requiring up to $10 \times$ fewer samples for unlearning. Additionally, under stronger MIA attacks on the CIFAR-100 dataset using a ResNet18 architecture, our approach outperforms the best baseline by $1.8\%$. Our code is available at https://github.com/sangamesh-kodge/class_forgetting.

Related papers

• LoRA Unlearns More and Retains More (Student Abstract) [0.0]
  PruneLoRA reduces the need for large-scale parameter updates by applying low-rank updates to the model. We leverage LoRA to selectively modify a subset of the pruned model's parameters, thereby reducing the computational cost, memory requirements and improving the model's ability to retain performance on the remaining classes.
  arXiv  Detail & Related papers  (2024-11-16T16:47:57Z)
• $\nabla τ$: Gradient-based and Task-Agnostic machine Unlearning [7.04736023670375]
  We introduce Gradient-based and Task-Agnostic machine Unlearning ($nabla tau$) $nabla tau$ applies adaptive gradient ascent to the data to be forgotten while using standard gradient descent for the remaining data. We evaluate our framework's effectiveness using a set of well-established Membership Inference Attack metrics.
  arXiv  Detail & Related papers  (2024-03-21T12:11:26Z)
• Class-Imbalanced Semi-Supervised Learning for Large-Scale Point Cloud Semantic Segmentation via Decoupling Optimization [64.36097398869774]
  Semi-supervised learning (SSL) has been an active research topic for large-scale 3D scene understanding. The existing SSL-based methods suffer from severe training bias due to class imbalance and long-tail distributions of the point cloud data. We introduce a new decoupling optimization framework, which disentangles feature representation learning and classifier in an alternative optimization manner to shift the bias decision boundary effectively.
  arXiv  Detail & Related papers  (2024-01-13T04:16:40Z)
• Boundary Unlearning [5.132489421775161]
  We propose Boundary Unlearning, a rapid yet effective way to unlearn an entire class from a trained machine learning model. We extensively evaluate Boundary Unlearning on image classification and face recognition tasks, with an expected speed-up of $17times$ and $19times$, respectively.
  arXiv  Detail & Related papers  (2023-03-21T03:33:18Z)
• One-Pixel Shortcut: on the Learning Preference of Deep Neural Networks [28.502489028888608]
  Unlearnable examples (ULEs) aim to protect data from unauthorized usage for training DNNs. In adversarial training, the unlearnability of error-minimizing noise will severely degrade. We propose a novel model-free method, named emphOne-Pixel Shortcut, which only perturbs a single pixel of each image and makes the dataset unlearnable.
  arXiv  Detail & Related papers  (2022-05-24T15:17:52Z)
• Spacing Loss for Discovering Novel Categories [72.52222295216062]
  Novel Class Discovery (NCD) is a learning paradigm, where a machine learning model is tasked to semantically group instances from unlabeled data. We first characterize existing NCD approaches into single-stage and two-stage methods based on whether they require access to labeled and unlabeled data together. We devise a simple yet powerful loss function that enforces separability in the latent space using cues from multi-dimensional scaling.
  arXiv  Detail & Related papers  (2022-04-22T09:37:11Z)
• Incremental Learning of Structured Memory via Closed-Loop Transcription [20.255633973040183]
  This work proposes a minimal computational model for learning structured memories of multiple object classes in an incremental setting. Our method is simpler than existing approaches for incremental learning, and more efficient in terms of model size, storage, and computation. Experimental results show that our method can effectively alleviate catastrophic forgetting, achieving significantly better performance than prior work of generative replay.
  arXiv  Detail & Related papers  (2022-02-11T02:20:43Z)
• Online Selective Classification with Limited Feedback [82.68009460301585]
  We study selective classification in the online learning model, wherein a predictor may abstain from classifying an instance. Two salient aspects of the setting we consider are that the data may be non-realisable, due to which abstention may be a valid long-term action. We construct simple versioning-based schemes for any $mu in (0,1],$ that make most $Tmu$ mistakes while incurring smash$tildeO(T1-mu)$ excess abstention against adaptive adversaries.
  arXiv  Detail & Related papers  (2021-10-27T08:00:53Z)
• Effective Model Sparsification by Scheduled Grow-and-Prune Methods [73.03533268740605]
  We propose a novel scheduled grow-and-prune (GaP) methodology without pre-training the dense models. Experiments have shown that such models can match or beat the quality of highly optimized dense models at 80% sparsity on a variety of tasks.
  arXiv  Detail & Related papers  (2021-06-18T01:03:13Z)
• Improving Calibration for Long-Tailed Recognition [68.32848696795519]
  We propose two methods to improve calibration and performance in such scenarios. For dataset bias due to different samplers, we propose shifted batch normalization. Our proposed methods set new records on multiple popular long-tailed recognition benchmark datasets.
  arXiv  Detail & Related papers  (2021-04-01T13:55:21Z)
• SCAN: Learning to Classify Images without Labels [73.69513783788622]
  We advocate a two-step approach where feature learning and clustering are decoupled. A self-supervised task from representation learning is employed to obtain semantically meaningful features. We obtain promising results on ImageNet, and outperform several semi-supervised learning methods in the low-data regime.
  arXiv  Detail & Related papers  (2020-05-25T18:12:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.