Leveraging Per-Instance Privacy for Machine Unlearning

• URL: http://arxiv.org/abs/2505.18786v1
• Date: Sat, 24 May 2025 16:55:57 GMT
• Title: Leveraging Per-Instance Privacy for Machine Unlearning
• Abstract summary: We present a principled, per-instance approach to quantifying the difficulty of unlearning via fine-tuning.<n>Our findings provide a foundation for more efficient and adaptive unlearning strategies tailored to the unique properties of individual data points.
• Score: 43.55855852029299
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present a principled, per-instance approach to quantifying the difficulty of unlearning via fine-tuning. We begin by sharpening an analysis of noisy gradient descent for unlearning (Chien et al., 2024), obtaining a better utility-unlearning tradeoff by replacing worst-case privacy loss bounds with per-instance privacy losses (Thudi et al., 2024), each of which bounds the (Renyi) divergence to retraining without an individual data point. To demonstrate the practical applicability of our theory, we present empirical results showing that our theoretical predictions are born out both for Stochastic Gradient Langevin Dynamics (SGLD) as well as for standard fine-tuning without explicit noise. We further demonstrate that per-instance privacy losses correlate well with several existing data difficulty metrics, while also identifying harder groups of data points, and introduce novel evaluation methods based on loss barriers. All together, our findings provide a foundation for more efficient and adaptive unlearning strategies tailored to the unique properties of individual data points.

Related papers

• Loss-Driven Bayesian Active Learning [14.594845964572663]
  We show how any loss can be used to derive a unique objective for optimal data acquisition.<n>In regression and classification experiments with a range of different losses, we find our approach reduces test losses relative to existing techniques.
  arXiv  Detail & Related papers  (2026-04-13T19:36:03Z)
• Certified Per-Instance Unlearning Using Individual Sensitivity Bounds [6.029433950934382]
  We investigate an alternative approach based on adaptive per-instance noise calibration tailored to the individual contribution of each data point to the learned solution.<n>For ridge regression trained via Langevin dynamics, we derive high-probability per-instance sensitivity bounds, yielding certified unlearning with substantially less noise injection.
  arXiv  Detail & Related papers  (2026-02-17T14:18:47Z)
• Machine Unlearning via Information Theoretic Regularization [3.05179671246628]
  We introduce a mathematical framework based on information-theoretic regularization to address both feature and data point unlearning.<n>By combining flexibility in learning objectives with simplicity in regularization design, our approach is highly adaptable and practical for a wide range of machine learning and AI applications.
  arXiv  Detail & Related papers  (2025-02-08T20:33:06Z)
• The Utility and Complexity of in- and out-of-Distribution Machine Unlearning [16.879887267565742]
  We analyze the fundamental utility, time, and space complexity trade-offs of approximate unlearning.<n>We propose a new robust and noisy gradient descent variant that provably amortizes unlearning time complexity without compromising utility.
  arXiv  Detail & Related papers  (2024-12-12T09:54:38Z)
• A Unified Generalization Analysis of Re-Weighting and Logit-Adjustment for Imbalanced Learning [129.63326990812234]
  We propose a technique named data-dependent contraction to capture how modified losses handle different classes. On top of this technique, a fine-grained generalization bound is established for imbalanced learning, which helps reveal the mystery of re-weighting and logit-adjustment.
  arXiv  Detail & Related papers  (2023-10-07T09:15:08Z)
• MaxMatch: Semi-Supervised Learning with Worst-Case Consistency [149.03760479533855]
  We propose a worst-case consistency regularization technique for semi-supervised learning (SSL) We present a generalization bound for SSL consisting of the empirical loss terms observed on labeled and unlabeled training data separately. Motivated by this bound, we derive an SSL objective that minimizes the largest inconsistency between an original unlabeled sample and its multiple augmented variants.
  arXiv  Detail & Related papers  (2022-09-26T12:04:49Z)
• Assaying Out-Of-Distribution Generalization in Transfer Learning [103.57862972967273]
  We take a unified view of previous work, highlighting message discrepancies that we address empirically. We fine-tune over 31k networks, from nine different architectures in the many- and few-shot setting.
  arXiv  Detail & Related papers  (2022-07-19T12:52:33Z)
• Learning to be adversarially robust and differentially private [42.7930886063265]
  We study the difficulties in learning that arise from robust and differentially private optimization. Data dimensionality dependent term introduced by private optimization compounds difficulties of learning a robust model. Size of adversarial generalization and clipping norm in differential privacy both increase the curvature of the loss landscape, implying poorer performance.
  arXiv  Detail & Related papers  (2022-01-06T22:33:06Z)
• On Covariate Shift of Latent Confounders in Imitation and Reinforcement Learning [69.48387059607387]
  We consider the problem of using expert data with unobserved confounders for imitation and reinforcement learning. We analyze the limitations of learning from confounded expert data with and without external reward. We validate our claims empirically on challenging assistive healthcare and recommender system simulation tasks.
  arXiv  Detail & Related papers  (2021-10-13T07:31:31Z)
• PEARL: Data Synthesis via Private Embeddings and Adversarial Reconstruction Learning [1.8692254863855962]
  We propose a new framework of data using deep generative models in a differentially private manner. Within our framework, sensitive data are sanitized with rigorous privacy guarantees in a one-shot fashion. Our proposal has theoretical guarantees of performance, and empirical evaluations on multiple datasets show that our approach outperforms other methods at reasonable levels of privacy.
  arXiv  Detail & Related papers  (2021-06-08T18:00:01Z)
• Risk Minimization from Adaptively Collected Data: Guarantees for Supervised and Policy Learning [57.88785630755165]
  Empirical risk minimization (ERM) is the workhorse of machine learning, but its model-agnostic guarantees can fail when we use adaptively collected data. We study a generic importance sampling weighted ERM algorithm for using adaptively collected data to minimize the average of a loss function over a hypothesis class. For policy learning, we provide rate-optimal regret guarantees that close an open gap in the existing literature whenever exploration decays to zero.
  arXiv  Detail & Related papers  (2021-06-03T09:50:13Z)
• Privacy-Preserving Federated Learning on Partitioned Attributes [6.661716208346423]
  Federated learning empowers collaborative training without exposing local data or models. We introduce an adversarial learning based procedure which tunes a local model to release privacy-preserving intermediate representations. To alleviate the accuracy decline, we propose a defense method based on the forward-backward splitting algorithm.
  arXiv  Detail & Related papers  (2021-04-29T14:49:14Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.