Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods

• URL: http://arxiv.org/abs/2508.12730v2
• Date: Tue, 28 Oct 2025 08:47:58 GMT
• Title: Unlearning Comparator: A Visual Analytics System for Comparative Evaluation of Machine Unlearning Methods
• Authors: Jaeung Lee, Suhyeon Yu, Yurim Jang, Simon S. Woo, Jaemin Jo,
• Abstract summary: Machine Unlearning (MU) aims to remove target training data from a trained model so that the removed data no longer influences the model's behavior.<n>Yet, researchers in this rapidly emerging field face challenges in analyzing and understanding the behavior of different MU methods.<n>We introduce a visual analytics system, Unlearning Comparator, designed to facilitate the systematic evaluation of MU methods.
• Score: 23.6050988823262
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Machine Unlearning (MU) aims to remove target training data from a trained model so that the removed data no longer influences the model's behavior, fulfilling "right to be forgotten" obligations under data privacy laws. Yet, we observe that researchers in this rapidly emerging field face challenges in analyzing and understanding the behavior of different MU methods, especially in terms of three fundamental principles in MU: accuracy, efficiency, and privacy. Consequently, they often rely on aggregate metrics and ad-hoc evaluations, making it difficult to accurately assess the trade-offs between methods. To fill this gap, we introduce a visual analytics system, Unlearning Comparator, designed to facilitate the systematic evaluation of MU methods. Our system supports two important tasks in the evaluation process: model comparison and attack simulation. First, it allows the user to compare the behaviors of two models, such as a model generated by a certain method and a retrained baseline, at class-, instance-, and layer-levels to better understand the changes made after unlearning. Second, our system simulates membership inference attacks (MIAs) to evaluate the privacy of a method, where an attacker attempts to determine whether specific data samples were part of the original training set. We evaluate our system through a case study visually analyzing prominent MU methods and demonstrate that it helps the user not only understand model behaviors but also gain insights that can inform the improvement of MU methods. The source code is publicly available at https://github.com/gnueaj/Machine-Unlearning-Comparator.

Related papers

• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• Towards Lifecycle Unlearning Commitment Management: Measuring Sample-level Unlearning Completeness [30.596695293390415]
  Interpolated Approximate Measurement (IAM) is a framework designed for unlearning inference.<n>IAM quantifies sample-level unlearning completeness by interpolating the model's generalization-fitting behavior gap on queried samples.<n>We apply IAM to recent approximate unlearning algorithms, revealing general risks of both over-unlearning and under-unlearning.
  arXiv  Detail & Related papers  (2025-06-06T14:22:18Z)
• MUBox: A Critical Evaluation Framework of Deep Machine Unlearning [13.186439491394474]
  MUBox is a comprehensive platform designed to evaluate unlearning methods in deep learning.<n> MUBox integrates 23 advanced unlearning techniques, tested across six practical scenarios with 11 diverse evaluation metrics.
  arXiv  Detail & Related papers  (2025-05-13T13:50:51Z)
• Instance-Level Data-Use Auditing of Visual ML Models [49.862257986549885]
  Growing trend of legal disputes over the unauthorized use of data in machine learning (ML) systems highlights the need for reliable data-use auditing mechanisms.<n>We present the first proactive, instance-level, data-use auditing method designed to enable data owners to audit the use of their individual data instances in ML models.
  arXiv  Detail & Related papers  (2025-03-28T13:28:57Z)
• Attribute-to-Delete: Machine Unlearning via Datamodel Matching [65.13151619119782]
  Machine unlearning -- efficiently removing a small "forget set" training data on a pre-divertrained machine learning model -- has recently attracted interest. Recent research shows that machine unlearning techniques do not hold up in such a challenging setting.
  arXiv  Detail & Related papers  (2024-10-30T17:20:10Z)
• Detecting Training Data of Large Language Models via Expectation Maximization [62.28028046993391]
  We introduce EM-MIA, a novel membership inference method that iteratively refines membership scores and prefix scores via an expectation-maximization algorithm.<n> EM-MIA achieves state-of-the-art results on WikiMIA.
  arXiv  Detail & Related papers  (2024-10-10T03:31:16Z)
• Deep Unlearn: Benchmarking Machine Unlearning for Image Classification [7.450700594277741]
  Machine unlearning (MU) aims to remove the influence of particular data points from the learnable parameters of a trained machine learning model.<n>This paper investigates 18 state-of-the-art MU methods across various benchmark datasets and models.
  arXiv  Detail & Related papers  (2024-10-02T06:41:58Z)
• Adversarial Machine Unlearning [26.809123658470693]
  This paper focuses on the challenge of machine unlearning, aiming to remove the influence of specific training data on machine learning models. Traditionally, the development of unlearning algorithms runs parallel with that of membership inference attacks (MIA), a type of privacy threat. We propose a game-theoretic framework that integrates MIAs into the design of unlearning algorithms.
  arXiv  Detail & Related papers  (2024-06-11T20:07:22Z)
• MUC: Machine Unlearning for Contrastive Learning with Black-box Evaluation [33.418062986773606]
  This paper introduces the Machine Unlearning for Contrastive Learning (MUC) framework and adapting existing methods.<n>We identify limitations in current approaches, noting that several methods perform inadequately as unlearners and that existing evaluation tools validate unlearning effects in contrastive learning.<n>We demonstrate that AC achieves state-of-the-art performance, approximating exact unlearning (retraining), and enables data owners to clearly visualize unlearning effects through black-box evaluation.
  arXiv  Detail & Related papers  (2024-06-05T19:55:45Z)
• An Information Theoretic Evaluation Metric For Strong Unlearning [20.143627174765985]
  We introduce the Information Difference Index (IDI), a novel white-box metric inspired by information theory. IDI quantifies retained information in intermediate features by measuring mutual information between those features and the labels to be forgotten. Our experiments demonstrate that IDI effectively measures the degree of unlearning across various datasets and architectures.
  arXiv  Detail & Related papers  (2024-05-28T06:57:01Z)
• Distilled Datamodel with Reverse Gradient Matching [74.75248610868685]
  We introduce an efficient framework for assessing data impact, comprising offline training and online evaluation stages. Our proposed method achieves comparable model behavior evaluation while significantly speeding up the process compared to the direct retraining method.
  arXiv  Detail & Related papers  (2024-04-22T09:16:14Z)
• A Probabilistic Model Behind Self-Supervised Learning [53.64989127914936]
  In self-supervised learning (SSL), representations are learned via an auxiliary task without annotated labels. We present a generative latent variable model for self-supervised learning. We show that several families of discriminative SSL, including contrastive methods, induce a comparable distribution over representations.
  arXiv  Detail & Related papers  (2024-02-02T13:31:17Z)
• Towards Better Modeling with Missing Data: A Contrastive Learning-based Visual Analytics Perspective [7.577040836988683]
  Missing data can pose a challenge for machine learning (ML) modeling. Current approaches are categorized into feature imputation and label prediction. This study proposes a Contrastive Learning framework to model observed data with missing values.
  arXiv  Detail & Related papers  (2023-09-18T13:16:24Z)
• Interpretable Multi-dataset Evaluation for Named Entity Recognition [110.64368106131062]
  We present a general methodology for interpretable evaluation for the named entity recognition (NER) task. The proposed evaluation method enables us to interpret the differences in models and datasets, as well as the interplay between them. By making our analysis tool available, we make it easy for future researchers to run similar analyses and drive progress in this area.
  arXiv  Detail & Related papers  (2020-11-13T10:53:27Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.