On Conformal Machine Unlearning

• URL: http://arxiv.org/abs/2508.03245v2
• Date: Wed, 01 Oct 2025 07:55:45 GMT
• Title: On Conformal Machine Unlearning
• Authors: Yahya Alkhatib, Wee Peng Tay,
• Abstract summary: We introduce a new definition for machine unlearning (MU) based on conformal prediction (CP)<n>We formalize the proposed conformal criteria that quantify how often forgotten samples are excluded from CP sets, and propose empirical metrics to measure the effectiveness of unlearning.
• Score: 23.735173540590832
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The increasing demand for data privacy has made machine unlearning (MU) essential for removing the influence of specific training samples from machine learning models while preserving performance on retained data. However, most existing MU methods lack rigorous statistical guarantees or rely on heuristic metrics such as accuracy. To overcome these limitations, we introduce a new definition for MU based on conformal prediction (CP), providing statistically sound, uncertainty-aware guarantees without the need for the concept of naive retraining. We formalize the proposed conformal criteria that quantify how often forgotten samples are excluded from CP sets, and propose empirical metrics to measure the effectiveness of unlearning. We further present a practical unlearning method designed to optimize these conformal metrics. Extensive experiments across diverse forgetting scenarios, datasets and models demonstrate the efficacy of our approach in removing targeted data.

Related papers

• Forgetting-MarI: LLM Unlearning via Marginal Information Regularization [6.979586479353831]
  Existing unlearning methods often degrade model performance by removing more information than necessary when attempting to ''forget'' specific data.<n>We introduce Forgetting-MarI, an LLM unlearning framework that provably removes only the additional (marginal) information contributed by the data to be unlearned.<n>By penalizing marginal information, our method yields an explicit upper bound on the unlearn dataset's residual influence in the trained models, providing provable undetectability.
  arXiv  Detail & Related papers  (2025-11-14T22:48:39Z)
• Uncertainty Quantification for Regression using Proper Scoring Rules [76.24649098854219]
  We introduce a unified UQ framework for regression based on proper scoring rules, such as CRPS, logarithmic, squared error, and quadratic scores.<n>We derive closed-form expressions for the uncertainty measures under practical parametric assumptions and show how to estimate them using ensembles of models.<n>Our broad evaluation on synthetic and real-world regression datasets provides guidance for selecting reliable UQ measures.
  arXiv  Detail & Related papers  (2025-09-30T17:52:12Z)
• GUARD: Guided Unlearning and Retention via Data Attribution for Large Language Models [23.667160042806064]
  GUARD is a novel framework for guided unlearning and retention via data attribution.<n>At its core, GUARD introduces a lightweight proxy data attribution metric tailored for LLM unlearning.<n>We provide rigorous theoretical guarantees that GUARD significantly enhances retention while maintaining forgetting metrics comparable to prior methods.
  arXiv  Detail & Related papers  (2025-06-12T17:49:09Z)
• Redefining Machine Unlearning: A Conformal Prediction-Motivated Approach [11.609354498110358]
  Machine unlearning seeks to remove the influence of specified data from a trained model.<n>In this paper, we find that the data misclassified across UA and MIA still have their ground truth labels included in the prediction set.<n>We propose two novel metrics inspired by conformal prediction that more reliably evaluate forgetting quality.
  arXiv  Detail & Related papers  (2025-01-31T18:58:43Z)
• Adaptive Conformal Inference by Betting [51.272991377903274]
  We consider the problem of adaptive conformal inference without any assumptions about the data generating process.<n>Existing approaches for adaptive conformal inference are based on optimizing the pinball loss using variants of online gradient descent.<n>We propose a different approach for adaptive conformal inference that leverages parameter-free online convex optimization techniques.
  arXiv  Detail & Related papers  (2024-12-26T18:42:08Z)
• Conformal Risk Minimization with Variance Reduction [37.74931189657469]
  Conformal prediction (CP) is a distribution-free framework for achieving probabilistic guarantees on black-box models.<n>Recent research efforts have focused on optimizing CP efficiency during training.<n>We formalize this concept as the problem of conformal risk minimization.
  arXiv  Detail & Related papers  (2024-11-03T21:48:15Z)
• Evaluating of Machine Unlearning: Robustness Verification Without Prior Modifications [15.257558809246524]
  Unlearning is a process enabling pre-trained models to remove the influence of specific training samples. Existing verification methods rely on machine learning attack techniques, such as membership inference attacks (MIAs) or backdoor attacks. We propose a novel verification scheme without any prior modifications, and can support verification on a much larger set.
  arXiv  Detail & Related papers  (2024-10-14T03:19:14Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Hessian-Free Online Certified Unlearning [8.875278412741695]
  We develop an online unlearning algorithm that achieves near-instantaneous data removal.<n>We prove that our proposed method outperforms the state-of-the-art methods in terms of the unlearning and generalization guarantees.
  arXiv  Detail & Related papers  (2024-04-02T07:54:18Z)
• Machine unlearning through fine-grained model parameters perturbation [26.653596302257057]
  We propose fine-grained Top-K and Random-k parameters perturbed inexact machine unlearning strategies.<n>We also tackle the challenge of evaluating the effectiveness of machine unlearning.
  arXiv  Detail & Related papers  (2024-01-09T07:14:45Z)
• Weak Supervision Performance Evaluation via Partial Identification [46.73061437177238]
  Programmatic Weak Supervision (PWS) enables supervised model training without direct access to ground truth labels. We present a novel method to address this challenge by framing model evaluation as a partial identification problem. Our approach derives reliable bounds on key metrics without requiring labeled data, overcoming core limitations in current weak supervision evaluation techniques.
  arXiv  Detail & Related papers  (2023-12-07T07:15:11Z)
• Stabilizing Subject Transfer in EEG Classification with Divergence Estimation [17.924276728038304]
  We propose several graphical models to describe an EEG classification task. We identify statistical relationships that should hold true in an idealized training scenario. We design regularization penalties to enforce these relationships in two stages.
  arXiv  Detail & Related papers  (2023-10-12T23:06:52Z)
• Conformal Prediction for Federated Uncertainty Quantification Under Label Shift [57.54977668978613]
  Federated Learning (FL) is a machine learning framework where many clients collaboratively train models. We develop a new conformal prediction method based on quantile regression and take into account privacy constraints.
  arXiv  Detail & Related papers  (2023-06-08T11:54:58Z)
• Federated Conformal Predictors for Distributed Uncertainty Quantification [83.50609351513886]
  Conformal prediction is emerging as a popular paradigm for providing rigorous uncertainty quantification in machine learning. In this paper, we extend conformal prediction to the federated learning setting. We propose a weaker notion of partial exchangeability, better suited to the FL setting, and use it to develop the Federated Conformal Prediction framework.
  arXiv  Detail & Related papers  (2023-05-27T19:57:27Z)
• Mitigating Catastrophic Forgetting in Task-Incremental Continual Learning with Adaptive Classification Criterion [50.03041373044267]
  We propose a Supervised Contrastive learning framework with adaptive classification criterion for Continual Learning. Experiments show that CFL achieves state-of-the-art performance and has a stronger ability to overcome compared with the classification baselines.
  arXiv  Detail & Related papers  (2023-05-20T19:22:40Z)
• Uncertainty Estimation by Fisher Information-based Evidential Deep Learning [61.94125052118442]
  Uncertainty estimation is a key factor that makes deep learning reliable in practical applications. We propose a novel method, Fisher Information-based Evidential Deep Learning ($mathcalI$-EDL) In particular, we introduce Fisher Information Matrix (FIM) to measure the informativeness of evidence carried by each sample, according to which we can dynamically reweight the objective loss terms to make the network more focused on the representation learning of uncertain classes.
  arXiv  Detail & Related papers  (2023-03-03T16:12:59Z)
• Improving Adaptive Conformal Prediction Using Self-Supervised Learning [72.2614468437919]
  We train an auxiliary model with a self-supervised pretext task on top of an existing predictive model and use the self-supervised error as an additional feature to estimate nonconformity scores. We empirically demonstrate the benefit of the additional information using both synthetic and real data on the efficiency (width), deficit, and excess of conformal prediction intervals.
  arXiv  Detail & Related papers  (2023-02-23T18:57:14Z)
• Post-hoc Uncertainty Learning using a Dirichlet Meta-Model [28.522673618527417]
  We propose a novel Bayesian meta-model to augment pre-trained models with better uncertainty quantification abilities. Our proposed method requires no additional training data and is flexible enough to quantify different uncertainties. We demonstrate our proposed meta-model approach's flexibility and superior empirical performance on these applications.
  arXiv  Detail & Related papers  (2022-12-14T17:34:11Z)
• Predictive machine learning for prescriptive applications: a coupled training-validating approach [77.34726150561087]
  We propose a new method for training predictive machine learning models for prescriptive applications. This approach is based on tweaking the validation step in the standard training-validating-testing scheme. Several experiments with synthetic data demonstrate promising results in reducing the prescription costs in both deterministic and real models.
  arXiv  Detail & Related papers  (2021-10-22T15:03:20Z)
• Uncertainty Weighted Actor-Critic for Offline Reinforcement Learning [63.53407136812255]
  Offline Reinforcement Learning promises to learn effective policies from previously-collected, static datasets without the need for exploration. Existing Q-learning and actor-critic based off-policy RL algorithms fail when bootstrapping from out-of-distribution (OOD) actions or states. We propose Uncertainty Weighted Actor-Critic (UWAC), an algorithm that detects OOD state-action pairs and down-weights their contribution in the training objectives accordingly.
  arXiv  Detail & Related papers  (2021-05-17T20:16:46Z)
• Training Keyword Spotting Models on Non-IID Data with Federated Learning [6.784774147680782]
  We show that a production-quality keyword-spotting model can be trained on-device using federated learning. To overcome the algorithmic constraints associated with fitting on-device data, we conduct thorough empirical studies of optimization algorithms. We label examples (given the zero visibility into on-device data) to explore teacher-student training.
  arXiv  Detail & Related papers  (2020-05-21T00:53:33Z)
• Meta-Learned Confidence for Few-shot Learning [60.6086305523402]
  A popular transductive inference technique for few-shot metric-based approaches, is to update the prototype of each class with the mean of the most confident query examples. We propose to meta-learn the confidence for each query sample, to assign optimal weights to unlabeled queries. We validate our few-shot learning model with meta-learned confidence on four benchmark datasets.
  arXiv  Detail & Related papers  (2020-02-27T10:22:17Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.