Unlearning Evaluation through Subset Statistical Independence

• URL: http://arxiv.org/abs/2603.00587v1
• Date: Sat, 28 Feb 2026 10:36:50 GMT
• Title: Unlearning Evaluation through Subset Statistical Independence
• Abstract summary: Most unlearning algorithms remove a small, random subset of the training data.<n>We propose a subset-level evaluation framework based on statistical independence.<n>Our method provides a simple, standalone evaluation procedure that aligns with unlearning.
• Score: 12.18235906179985
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Evaluating machine unlearning remains challenging, as existing methods typically require retraining reference models or performing membership inference attacks, both of which rely on prior access to training configuration or supervision labels, making them impractical in realistic scenarios. Motivated by the fact that most unlearning algorithms remove a small, random subset of the training data, we propose a subset-level evaluation framework based on statistical independence. Specifically, we design a tailored use of the Hilbert-Schmidt Independence Criterion to assess whether the model outputs on a given subset exhibit statistical dependence, without requiring model retraining or auxiliary classifiers. Our method provides a simple, standalone evaluation procedure that aligns with unlearning workflows. Extensive experiments demonstrate that our approach reliably distinguishes in-training from out-of-training subsets and clearly differentiates unlearning effectiveness, even when existing evaluations fall short.

Related papers

• Auditing Machine Unlearning: A Systematic Research on Whether Models Truly Forget [25.03537230361785]
  We propose the first practical and general-purpose auditing framework for machine unlearning.<n>Our framework addresses the key limitations of existing methods by eliminating the need for retraining-from-scratch baselines.<n>We show that our framework is robust against fake unlearning attempts and generalizes well to large language models.
  arXiv  Detail & Related papers  (2026-06-15T01:56:09Z)
• Not Every Subject Should Stay: Machine Unlearning for Noisy Engagement Recognition [53.005382593686356]
  Engagement recognition datasets are typically subject-indexed and often contain noisy, subjective supervision.<n>We study this setting through subject-level machine unlearning as a post-hoc sanitization mechanism for engagement recognition.
  arXiv  Detail & Related papers  (2026-05-06T10:03:06Z)
• Evaluating Supervised Machine Learning Models: Principles, Pitfalls, and Metric Selection [5.015895904388096]
  This paper examines the principles, challenges, and practical considerations involved in evaluating supervised learning algorithms.<n>It discusses how evaluation outcomes are influenced by dataset characteristics, validation design, class imbalance, asymmetric error costs, and the choice of performance metrics.
  arXiv  Detail & Related papers  (2026-04-15T13:44:35Z)
• Inference-time Unlearning Using Conformal Prediction [13.479885316485209]
  Unlearning is the process of efficiently removing specific information from a trained machine learning model without retraining from scratch.<n>This paper introduces a framework that iteratively refines the quality of the generated responses using feedback from the verifier without updating the model parameters.<n>This paper's approach significantly outperforms existing state-of-the-art methods, reducing unlearning error by up to 93% across challenging unlearning benchmarks.
  arXiv  Detail & Related papers  (2026-02-03T17:46:50Z)
• Forgetting Similar Samples: Can Machine Unlearning Do it Better? [35.26900233614191]
  We argue that machine unlearning methods mainly aim at removing samples rather than removing samples' influence on the model.<n>We conduct a study to evaluate the effectiveness of existing unlearning schemes when the training dataset includes many samples similar to those targeted for unlearning.<n>Our experiments, conducted on four carefully constructed datasets with thorough analysis, reveal a notable gap between the expected and actual performance of most existing unlearning methods.
  arXiv  Detail & Related papers  (2026-01-11T14:55:57Z)
• Reference-Specific Unlearning Metrics Can Hide the Truth: A Reality Check [60.77691669644931]
  We propose Functional Alignment for Distributional Equivalence (FADE), a novel metric that measures distributional similarity between unlearned and reference models.<n>We show that FADE captures functional alignment across the entire output distribution, providing a principled assessment of genuine unlearning.<n>These findings expose fundamental gaps in current evaluation practices and demonstrate that FADE provides a more robust foundation for developing and assessing truly effective unlearning methods.
  arXiv  Detail & Related papers  (2025-10-14T20:50:30Z)
• Rectifying Privacy and Efficacy Measurements in Machine Unlearning: A New Inference Attack Perspective [42.003102851493885]
  We propose RULI (Rectified Unlearning Evaluation Framework via Likelihood Inference) to address critical gaps in the evaluation of inexact unlearning methods.<n>RULI introduces a dual-objective attack to measure both unlearning efficacy and privacy risks at a per-sample granularity.<n>Our findings reveal significant vulnerabilities in state-of-the-art unlearning methods, exposing privacy risks underestimated by existing methods.
  arXiv  Detail & Related papers  (2025-06-16T00:30:02Z)
• Are We Truly Forgetting? A Critical Re-examination of Machine Unlearning Evaluation Protocols [14.961054239793356]
  We introduce a rigorous unlearning evaluation setup, in which forgetting classes exhibit semantic similarity to downstream task classes.<n>We hope our benchmark serves as a standardized protocol for evaluating unlearning algorithms under realistic conditions.
  arXiv  Detail & Related papers  (2025-03-10T07:11:34Z)
• Model-free Methods for Event History Analysis and Efficient Adjustment (PhD Thesis) [55.2480439325792]
  This thesis is a series of independent contributions to statistics unified by a model-free perspective.<n>The first chapter elaborates on how a model-free perspective can be used to formulate flexible methods that leverage prediction techniques from machine learning.<n>The second chapter studies the concept of local independence, which describes whether the evolution of one process is directly influenced by another.
  arXiv  Detail & Related papers  (2025-02-11T19:24:09Z)
• Evaluation of Seismic Artificial Intelligence with Uncertainty [0.0]
  We develop an evaluation framework for evaluating and comparing deep learning models (DLMs)<n>Our framework helps practitioners choose the best model for their problem and set performance expectations.
  arXiv  Detail & Related papers  (2025-01-15T16:45:51Z)
• Towards Effective Evaluations and Comparisons for LLM Unlearning Methods [97.2995389188179]
  This paper seeks to refine the evaluation of machine unlearning for large language models.<n>It addresses two key challenges -- the robustness of evaluation metrics and the trade-offs between competing goals.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Evaluating Disentanglement in Generative Models Without Knowledge of Latent Factors [71.79984112148865]
  We introduce a method for ranking generative models based on the training dynamics exhibited during learning. Inspired by recent theoretical characterizations of disentanglement, our method does not require supervision of the underlying latent factors.
  arXiv  Detail & Related papers  (2022-10-04T17:27:29Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.