SEMU: Singular Value Decomposition for Efficient Machine Unlearning

• URL: http://arxiv.org/abs/2502.07587v1
• Date: Tue, 11 Feb 2025 14:36:39 GMT
• Title: SEMU: Singular Value Decomposition for Efficient Machine Unlearning
• Authors: Marcin Sendera, Łukasz Struski, Kamil Książek, Kryspin Musiol, Jacek Tabor, Dawid Rymarczyk,
• Abstract summary: Machine unlearning (MU) has become increasingly critical to meet upcoming safety regulations. We propose Singular Value Decomposition for Efficient Machine Unlearning (SEMU) SEMU minimizes the number of model parameters that need to be modified, effectively removing unwanted knowledge.
• Score: 9.61813564612515
• License:
• Abstract: While the capabilities of generative foundational models have advanced rapidly in recent years, methods to prevent harmful and unsafe behaviors remain underdeveloped. Among the pressing challenges in AI safety, machine unlearning (MU) has become increasingly critical to meet upcoming safety regulations. Most existing MU approaches focus on altering the most significant parameters of the model. However, these methods often require fine-tuning substantial portions of the model, resulting in high computational costs and training instabilities, which are typically mitigated by access to the original training dataset. In this work, we address these limitations by leveraging Singular Value Decomposition (SVD) to create a compact, low-dimensional projection that enables the selective forgetting of specific data points. We propose Singular Value Decomposition for Efficient Machine Unlearning (SEMU), a novel approach designed to optimize MU in two key aspects. First, SEMU minimizes the number of model parameters that need to be modified, effectively removing unwanted knowledge while making only minimal changes to the model's weights. Second, SEMU eliminates the dependency on the original training dataset, preserving the model's previously acquired knowledge without additional data requirements. Extensive experiments demonstrate that SEMU achieves competitive performance while significantly improving efficiency in terms of both data usage and the number of modified parameters.

Related papers

• Attribute-to-Delete: Machine Unlearning via Datamodel Matching [65.13151619119782]
  Machine unlearning -- efficiently removing a small "forget set" training data on a pre-divertrained machine learning model -- has recently attracted interest. Recent research shows that machine unlearning techniques do not hold up in such a challenging setting.
  arXiv  Detail & Related papers  (2024-10-30T17:20:10Z)
• Deep Unlearn: Benchmarking Machine Unlearning [7.450700594277741]
  Machine unlearning (MU) aims to remove the influence of particular data points from the learnable parameters of a trained machine learning model. This paper investigates 18 state-of-the-art MU methods across various benchmark datasets and models.
  arXiv  Detail & Related papers  (2024-10-02T06:41:58Z)
• Towards Robust and Cost-Efficient Knowledge Unlearning for Large Language Models [25.91643745340183]
  Large Language Models (LLMs) have demonstrated strong reasoning and memorization capabilities via pretraining on massive textual corpora. This poses risk of privacy and copyright violations, highlighting the need for efficient machine unlearning methods. We propose two novel techniques for robust and efficient unlearning for LLMs.
  arXiv  Detail & Related papers  (2024-08-13T04:18:32Z)
• Learning to Unlearn for Robust Machine Unlearning [6.488418950340473]
  We introduce a novel Learning-to-Unlearn (LTU) framework to optimize the unlearning process. LTU includes a meta-optimization scheme that facilitates models to effectively preserve generalizable knowledge. We also introduce a Gradient Harmonization strategy to align the optimization trajectories for remembering and forgetting.
  arXiv  Detail & Related papers  (2024-07-15T07:36:00Z)
• EsaCL: Efficient Continual Learning of Sparse Models [10.227171407348326]
  Key challenge in the continual learning setting is to efficiently learn a sequence of tasks without forgetting how to perform previously learned tasks. We propose a new method for efficient continual learning of sparse models (EsaCL) that can automatically prune redundant parameters without adversely impacting the model's predictive power.
  arXiv  Detail & Related papers  (2024-01-11T04:59:44Z)
• Uncertainty-aware Parameter-Efficient Self-training for Semi-supervised Language Understanding [38.11411155621616]
  We study self-training as one of the predominant semi-supervised learning approaches. We present UPET, a novel Uncertainty-aware self-Training framework. We show that UPET achieves a substantial improvement in terms of performance and efficiency.
  arXiv  Detail & Related papers  (2023-10-19T02:18:29Z)
• Towards Robust Continual Learning with Bayesian Adaptive Moment Regularization [51.34904967046097]
  Continual learning seeks to overcome the challenge of catastrophic forgetting, where a model forgets previously learnt information. We introduce a novel prior-based method that better constrains parameter growth, reducing catastrophic forgetting. Results show that BAdam achieves state-of-the-art performance for prior-based methods on challenging single-headed class-incremental experiments.
  arXiv  Detail & Related papers  (2023-09-15T17:10:51Z)
• Value function estimation using conditional diffusion models for control [62.27184818047923]
  We propose a simple algorithm called Diffused Value Function (DVF) It learns a joint multi-step model of the environment-robot interaction dynamics using a diffusion model. We show how DVF can be used to efficiently capture the state visitation measure for multiple controllers.
  arXiv  Detail & Related papers  (2023-06-09T18:40:55Z)
• Model-Agnostic Multitask Fine-tuning for Few-shot Vision-Language Transfer Learning [59.38343286807997]
  We propose Model-Agnostic Multitask Fine-tuning (MAMF) for vision-language models on unseen tasks. Compared with model-agnostic meta-learning (MAML), MAMF discards the bi-level optimization and uses only first-order gradients. We show that MAMF consistently outperforms the classical fine-tuning method for few-shot transfer learning on five benchmark datasets.
  arXiv  Detail & Related papers  (2022-03-09T17:26:53Z)
• Hyperparameter-free Continuous Learning for Domain Classification in Natural Language Understanding [60.226644697970116]
  Domain classification is the fundamental task in natural language understanding (NLU) Most existing continual learning approaches suffer from low accuracy and performance fluctuation. We propose a hyper parameter-free continual learning model for text data that can stably produce high performance under various environments.
  arXiv  Detail & Related papers  (2022-01-05T02:46:16Z)
• Discrete Auto-regressive Variational Attention Models for Text Modeling [53.38382932162732]
  Variational autoencoders (VAEs) have been widely applied for text modeling. They are troubled by two challenges: information underrepresentation and posterior collapse. We propose Discrete Auto-regressive Variational Attention Model (DAVAM) to address the challenges.
  arXiv  Detail & Related papers  (2021-06-16T06:36:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.