How Does Overparameterization Affect Machine Unlearning of Deep Neural Networks?

• URL: http://arxiv.org/abs/2503.08633v1
• Date: Tue, 11 Mar 2025 17:21:26 GMT
• Title: How Does Overparameterization Affect Machine Unlearning of Deep Neural Networks?
• Authors: Gal Alon, Yehuda Dar,
• Abstract summary: We show how unlearning of deep neural networks (DNNs) is affected by the model parameterization level.<n>We define validation-based tuning for several unlearning methods from the recent literature.
• Score: 1.573034584191491
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning is the task of updating a trained model to forget specific training data without retraining from scratch. In this paper, we investigate how unlearning of deep neural networks (DNNs) is affected by the model parameterization level, which corresponds here to the DNN width. We define validation-based tuning for several unlearning methods from the recent literature, and show how these methods perform differently depending on (i) the DNN parameterization level, (ii) the unlearning goal (unlearned data privacy or bias removal), (iii) whether the unlearning method explicitly uses the unlearned examples. Our results show that unlearning excels on overparameterized models, in terms of balancing between generalization and achieving the unlearning goal; although for bias removal this requires the unlearning method to use the unlearned examples. We further elucidate our error-based analysis by measuring how much the unlearning changes the classification decision regions in the proximity of the unlearned examples, and avoids changing them elsewhere. By this we show that the unlearning success for overparameterized models stems from the ability to delicately change the model functionality in small regions in the input space while keeping much of the model functionality unchanged.

Related papers

• Sharpness-Aware Parameter Selection for Machine Unlearning [6.397490580631141]
  It often happens that some sensitive personal information, such as credit card numbers or passwords, are mistakenly incorporated in the training of machine learning models and need to be removed afterwards. There have been various machine unlearning techniques proposed in the literature to address this problem. Most of the proposed methods revolve around removing individual data samples from a trained model. While the existing methods for these tasks do the unlearning task by updating the whole set of model parameters or only the last layer of the model, we show that there are a subset of model parameters that have the largest contribution in the unlearning target features.
  arXiv  Detail & Related papers  (2025-04-08T19:41:07Z)
• Attribute-to-Delete: Machine Unlearning via Datamodel Matching [65.13151619119782]
  Machine unlearning -- efficiently removing a small "forget set" training data on a pre-divertrained machine learning model -- has recently attracted interest. Recent research shows that machine unlearning techniques do not hold up in such a challenging setting.
  arXiv  Detail & Related papers  (2024-10-30T17:20:10Z)
• Transferable Post-training via Inverse Value Learning [83.75002867411263]
  We propose modeling changes at the logits level during post-training using a separate neural network (i.e., the value network) After training this network on a small base model using demonstrations, this network can be seamlessly integrated with other pre-trained models during inference. We demonstrate that the resulting value network has broad transferability across pre-trained models of different parameter sizes.
  arXiv  Detail & Related papers  (2024-10-28T13:48:43Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• In-Context Unlearning: Language Models as Few Shot Unlearners [27.962361828354716]
  We propose a new class of unlearning methods for Large Language Models (LLMs) This method unlearns instances from the model by simply providing specific kinds of inputs in context, without the need to update model parameters. Our experimental results demonstrate that in-context unlearning performs on par with, or in some cases outperforms other state-of-the-art methods that require access to model parameters.
  arXiv  Detail & Related papers  (2023-10-11T15:19:31Z)
• Theoretical Characterization of the Generalization Performance of Overfitted Meta-Learning [70.52689048213398]
  This paper studies the performance of overfitted meta-learning under a linear regression model with Gaussian features. We find new and interesting properties that do not exist in single-task linear regression. Our analysis suggests that benign overfitting is more significant and easier to observe when the noise and the diversity/fluctuation of the ground truth of each training task are large.
  arXiv  Detail & Related papers  (2023-04-09T20:36:13Z)
• Adversarial Learning Networks: Source-free Unsupervised Domain Incremental Learning [0.0]
  In a non-stationary environment, updating a DNN model requires parameter re-training or model fine-tuning. We propose an unsupervised source-free method to update DNN classification models. Unlike existing methods, our approach can update a DNN model incrementally for non-stationary source and target tasks without storing past training data.
  arXiv  Detail & Related papers  (2023-01-28T02:16:13Z)
• Learning to Learn with Generative Models of Neural Network Checkpoints [71.06722933442956]
  We construct a dataset of neural network checkpoints and train a generative model on the parameters. We find that our approach successfully generates parameters for a wide range of loss prompts. We apply our method to different neural network architectures and tasks in supervised and reinforcement learning.
  arXiv  Detail & Related papers  (2022-09-26T17:59:58Z)
• Unrolling SGD: Understanding Factors Influencing Machine Unlearning [17.6607904333012]
  Machine unlearning is the process through which a deployed machine learning model forgets about one of its training data points. We first taxonomize approaches and metrics of approximate unlearning. We identify verification error, i.e., the L2 difference between the weights of an approximately unlearned and a naively retrained model.
  arXiv  Detail & Related papers  (2021-09-27T23:46:59Z)
• Machine Unlearning of Features and Labels [72.81914952849334]
  We propose first scenarios for unlearning and labels in machine learning models. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
  arXiv  Detail & Related papers  (2021-08-26T04:42:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.