Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions

• URL: http://arxiv.org/abs/2409.09778v3
• Date: Fri, 07 Feb 2025 22:19:33 GMT
• Title: Rewind-to-Delete: Certified Machine Unlearning for Nonconvex Functions
• Authors: Siqiao Mu, Diego Klabjan,
• Abstract summary: Machine unlearning algorithms aim to efficiently data from a model without it from scratch. Certified machine unlearning is a strong theoretical guarantee based on differential privacy.
• Score: 11.955062839855334
• License:
• Abstract: Machine unlearning algorithms aim to efficiently remove data from a model without retraining it from scratch, in order to remove corrupted or outdated data or respect a user's ``right to be forgotten." Certified machine unlearning is a strong theoretical guarantee based on differential privacy that quantifies the extent to which an algorithm erases data from the model weights. In contrast to existing works in certified unlearning for convex or strongly convex loss functions, or nonconvex objectives with limiting assumptions, we propose the first, first-order, black-box (i.e., can be applied to models pretrained with vanilla gradient descent) algorithm for unlearning on general nonconvex loss functions, which unlearns by ``rewinding" to an earlier step during the learning process before performing gradient descent on the loss function of the retained data points. We prove $(\epsilon, \delta)$ certified unlearning and performance guarantees that establish the privacy-utility-complexity tradeoff of our algorithm, and we prove generalization guarantees for nonconvex functions that satisfy the Polyak-Lojasiewicz inequality. Finally, we implement our algorithm under a new experimental framework that more accurately reflects real-world use cases for preserving user privacy.

Related papers

• Accelerated zero-order SGD under high-order smoothness and overparameterized regime [79.85163929026146]
  We present a novel gradient-free algorithm to solve convex optimization problems. Such problems are encountered in medicine, physics, and machine learning. We provide convergence guarantees for the proposed algorithm under both types of noise.
  arXiv  Detail & Related papers  (2024-11-21T10:26:17Z)
• LEARN: An Invex Loss for Outlier Oblivious Robust Online Optimization [56.67706781191521]
  An adversary can introduce outliers by corrupting loss functions in an arbitrary number of k, unknown to the learner. We present a robust online rounds optimization framework, where an adversary can introduce outliers by corrupting loss functions in an arbitrary number of k, unknown.
  arXiv  Detail & Related papers  (2024-08-12T17:08:31Z)
• Regression under demographic parity constraints via unlabeled post-processing [5.762345156477737]
  We present a general-purpose post-processing algorithm that generates predictions that meet the demographic parity. Unlike prior methods, our approach is fully theory-driven. We require precise control over the gradient norm of the convex function. Our algorithm is backed by finite-sample analysis and post-processing bounds, with experimental results validating our theoretical findings.
  arXiv  Detail & Related papers  (2024-07-22T08:11:58Z)
• Hessian-Free Online Certified Unlearning [8.875278412741695]
  We develop an online unlearning algorithm that achieves near-instantaneous data removal. We prove that our proposed method outperforms the state-of-the-art methods in terms of the unlearning and generalization guarantees.
  arXiv  Detail & Related papers  (2024-04-02T07:54:18Z)
• Certified Machine Unlearning via Noisy Stochastic Gradient Descent [20.546589699647416]
  Machine unlearning aims to efficiently remove the effect of certain data points on the trained model. We propose to leverage noisy gradient descent for unlearning and establish its first approximate unlearning guarantee.
  arXiv  Detail & Related papers  (2024-03-25T18:43:58Z)
• Loss-Free Machine Unlearning [51.34904967046097]
  We present a machine unlearning approach that is both retraining- and label-free. Retraining-free approaches often utilise Fisher information, which is derived from the loss and requires labelled data which may not be available. We present an extension to the Selective Synaptic Dampening algorithm, substituting the diagonal of the Fisher information matrix for the gradient of the l2 norm of the model output to approximate sensitivity.
  arXiv  Detail & Related papers  (2024-02-29T16:15:34Z)
• Equation Discovery with Bayesian Spike-and-Slab Priors and Efficient Kernels [57.46832672991433]
  We propose a novel equation discovery method based on Kernel learning and BAyesian Spike-and-Slab priors (KBASS) We use kernel regression to estimate the target function, which is flexible, expressive, and more robust to data sparsity and noises. We develop an expectation-propagation expectation-maximization algorithm for efficient posterior inference and function estimation.
  arXiv  Detail & Related papers  (2023-10-09T03:55:09Z)
• Forget Unlearning: Towards True Data-Deletion in Machine Learning [18.656957502454592]
  We show that unlearning is not equivalent to data deletion and does not guarantee the "right to be forgotten" We propose an accurate, computationally efficient, and secure data-deletion machine learning algorithm in the online setting.
  arXiv  Detail & Related papers  (2022-10-17T10:06:11Z)
• Algorithms that Approximate Data Removal: New Results and Limitations [2.6905021039717987]
  We study the problem of deleting user data from machine learning models trained using empirical risk minimization. We develop an online unlearning algorithm that is both computationally and memory efficient.
  arXiv  Detail & Related papers  (2022-09-25T17:20:33Z)
• Machine Unlearning of Features and Labels [72.81914952849334]
  We propose first scenarios for unlearning and labels in machine learning models. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
  arXiv  Detail & Related papers  (2021-08-26T04:42:24Z)
• Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning [57.88785630755165]
  Empirical risk minimization (ERM) is the workhorse of machine learning, but its model-agnostic guarantees can fail when we use adaptively collected data. We study a generic importance sampling weighted ERM algorithm for using adaptively collected data to minimize the average of a loss function over a hypothesis class. For policy learning, we provide rate-optimal regret guarantees that close an open gap in the existing literature whenever exploration decays to zero.
  arXiv  Detail & Related papers  (2021-06-03T09:50:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.