Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization

• URL: http://arxiv.org/abs/2503.02312v1
• Date: Tue, 04 Mar 2025 06:14:33 GMT
• Title: Go Beyond Your Means: Unlearning with Per-Sample Gradient Orthogonalization
• Authors: Aviv Shamsian, Eitan Shaar, Aviv Navon, Gal Chechik, Ethan Fetaya,
• Abstract summary: Machine unlearning aims to remove the influence of problematic training data after a model has been trained.<n>Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the unlearn data with the gradient descent on a retain set representing the training data.<n>Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and the retain set rather than competing ascent and descent processes.
• Score: 43.436621884831276
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning aims to remove the influence of problematic training data after a model has been trained. The primary challenge in machine unlearning is ensuring that the process effectively removes specified data without compromising the model's overall performance on the remaining dataset. Many existing machine unlearning methods address this challenge by carefully balancing gradient ascent on the unlearn data with the gradient descent on a retain set representing the training data. Here, we propose OrthoGrad, a novel approach that mitigates interference between the unlearn set and the retain set rather than competing ascent and descent processes. Our method projects the gradient of the unlearn set onto the subspace orthogonal to all gradients in the retain batch, effectively avoiding any gradient interference. We demonstrate the effectiveness of OrthoGrad on multiple machine unlearning benchmarks, including automatic speech recognition, outperforming competing methods.

Related papers

• Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions [11.955062839855334]
  Machine unlearning algorithms aim to efficiently data from a model without it from scratch.<n> Certified machine unlearning is a strong theoretical guarantee based on differential privacy.
  arXiv  Detail & Related papers  (2024-09-15T15:58:08Z)
• An Effective Dynamic Gradient Calibration Method for Continual Learning [11.555822066922508]
  Continual learning (CL) is a fundamental topic in machine learning, where the goal is to train a model with continuously incoming data and tasks. Due to the memory limit, we cannot store all the historical data, and therefore confront the catastrophic forgetting'' problem. We develop an effective algorithm to calibrate the gradient in each updating step of the model.
  arXiv  Detail & Related papers  (2024-07-30T16:30:09Z)
• Machine Unlearning via Null Space Calibration [23.668928015009087]
  We introduce machine underlineUnlearning via underlineNull underlineSpace underlineCalibration (UNSC), which can unlearn target samples without over-unlearning. Our approach hinges on confining the unlearning process to a specified null space tailored to the remaining samples.
  arXiv  Detail & Related papers  (2024-04-21T09:09:21Z)
• Incremental Self-training for Semi-supervised Learning [56.57057576885672]
  IST is simple yet effective and fits existing self-training-based semi-supervised learning methods. We verify the proposed IST on five datasets and two types of backbone, effectively improving the recognition accuracy and learning speed.
  arXiv  Detail & Related papers  (2024-04-14T05:02:00Z)
• Certified Machine Unlearning via Noisy Stochastic Gradient Descent [20.546589699647416]
  Machine unlearning aims to efficiently remove the effect of certain data points on the trained model.<n>We propose to leverage noisy gradient descent for unlearning and establish its first approximate unlearning guarantee.
  arXiv  Detail & Related papers  (2024-03-25T18:43:58Z)
• Loss-Free Machine Unlearning [51.34904967046097]
  We present a machine unlearning approach that is both retraining- and label-free. Retraining-free approaches often utilise Fisher information, which is derived from the loss and requires labelled data which may not be available. We present an extension to the Selective Synaptic Dampening algorithm, substituting the diagonal of the Fisher information matrix for the gradient of the l2 norm of the model output to approximate sensitivity.
  arXiv  Detail & Related papers  (2024-02-29T16:15:34Z)
• Dataset Condensation Driven Machine Unlearning [0.0]
  Current trend in data regulation requirements and privacy-preserving machine learning has emphasized the importance of machine unlearning. We propose new dataset condensation techniques and an innovative unlearning scheme that strikes a balance between machine unlearning privacy, utility, and efficiency. We present a novel and effective approach to instrumenting machine unlearning and propose its application in defending against membership inference and model inversion attacks.
  arXiv  Detail & Related papers  (2024-01-31T21:48:25Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• Minimizing the Accumulated Trajectory Error to Improve Dataset Distillation [151.70234052015948]
  We propose a novel approach that encourages the optimization algorithm to seek a flat trajectory. We show that the weights trained on synthetic data are robust against the accumulated errors perturbations with the regularization towards the flat trajectory. Our method, called Flat Trajectory Distillation (FTD), is shown to boost the performance of gradient-matching methods by up to 4.7%.
  arXiv  Detail & Related papers  (2022-11-20T15:49:11Z)
• Online Coreset Selection for Rehearsal-based Continual Learning [65.85595842458882]
  In continual learning, we store a subset of training examples (coreset) to be replayed later to alleviate catastrophic forgetting. We propose Online Coreset Selection (OCS), a simple yet effective method that selects the most representative and informative coreset at each iteration. Our proposed method maximizes the model's adaptation to a target dataset while selecting high-affinity samples to past tasks, which directly inhibits catastrophic forgetting.
  arXiv  Detail & Related papers  (2021-06-02T11:39:25Z)
• Attentional-Biased Stochastic Gradient Descent [74.49926199036481]
  We present a provable method (named ABSGD) for addressing the data imbalance or label noise problem in deep learning. Our method is a simple modification to momentum SGD where we assign an individual importance weight to each sample in the mini-batch. ABSGD is flexible enough to combine with other robust losses without any additional cost.
  arXiv  Detail & Related papers  (2020-12-13T03:41:52Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.