RULER: Representation-Level Verification of Machine Unlearning

• URL: http://arxiv.org/abs/2605.27569v2
• Date: Sun, 31 May 2026 21:06:35 GMT
• Title: RULER: Representation-Level Verification of Machine Unlearning
• Abstract summary: We introduce RULER, a set of representation-level verification metrics.<n>M2 measures whether forget-set records occupy the same representational position as in a model retrained without them.<n>M4 detects residuals from the unlearned model's internal similarity structure alone, without retraining.
• Score: 0.17188280334580197
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning aims to remove the influence of specific training records from a deployed model without retraining from scratch. Current protocols verify this at the output level through membership inference, retain accuracy, and forget-set accuracy, but a model can satisfy all three whilst still encoding forgotten records in its intermediate representations. We introduce RULER, a set of representation-level verification metrics. The oracle-comparative metric M2 measures whether forget-set records occupy the same representational position as in a model retrained without them. The oracle-free metric M4 detects residuals from the unlearned model's internal similarity structure alone, without retraining. Four approximate unlearning methods all pass output-level evaluation, yet under a linear mixed-effects model M2 detects significant residuals in 10 of 12 conditions (p<0.05), with effect sizes growing as the forget fraction increases. A fifth method, Bad Teacher, shows the same residuals despite a different forgetting mechanism. M4 acts as a pre-unlearning diagnostic across tabular, image, clinical text, and face-identity settings: it detects identity-level memorisation in face recognition models where no tested method fully erases the signal.

Related papers

• The BatchNorm Illusion: Diagnosing Normalization Artifacts in Machine Unlearning Evaluation [8.644777648927427]
  We identify a previously undocumented confound in how unlearning is evaluated on BatchNorm-based architectures.<n>We formalize this operation as a weight-preserving fixed-point operator and prove that any pre-versus-post gap it induces is provably attributable to BN running statistics.
  arXiv  Detail & Related papers  (2026-09-08T15:39:10Z)
• DriftScope: Measuring The Hidden Effects of Diffusion Model Adaptation [50.584834483403675]
  Adapting pre-trained text-to-image diffusion models, whether to learn new visual concepts or erase unwanted ones, is routinely evaluated on its intended effects alone.<n>We demonstrate that adaptation systematically damages semantically unrelated concepts in ways that aggregate metrics structurally cannot surface.<n>We introduce DriftScope, a prompt-level diagnostic tool that takes any two model checkpoints and returns a ranked list of tokens whose visual concepts have shifted most between them.
  arXiv  Detail & Related papers  (2026-06-30T20:57:51Z)
• Forgetting That Sticks: Quantization-Permanent Unlearning via Circuit Attribution [3.6704226968275253]
  We show gradient-based methods that achieve meaningful forgetting lose it under compression, while methods that survive quantization barely change the model.<n>We present MANSU, which resolves both modes by combining causal circuit attribution to isolate the minimal forget-set subgraph.<n>We additionally introduce Circuit Attribution Divergence (CAD), a mechanistic verification metric distinguishing structural erasure from behavioral suppression.
  arXiv  Detail & Related papers  (2026-05-14T17:44:10Z)
• Not Every Subject Should Stay: Machine Unlearning for Noisy Engagement Recognition [53.005382593686356]
  Engagement recognition datasets are typically subject-indexed and often contain noisy, subjective supervision.<n>We study this setting through subject-level machine unlearning as a post-hoc sanitization mechanism for engagement recognition.
  arXiv  Detail & Related papers  (2026-05-06T10:03:06Z)
• Learning the Signature of Memorization in Autoregressive Language Models [3.6048665052465663]
  We introduce the first transferable learned attack, enabled by the observation that fine-tuning any model on any corpus yields unlimited labeled data.<n>This removes the shadow model bottleneck and brings membership inference into the deep learning era.
  arXiv  Detail & Related papers  (2026-04-03T17:17:51Z)
• REMIND: Input Loss Landscapes Reveal Residual Memorization in Post-Unlearning LLMs [0.1784233255402269]
  Machine unlearning aims to remove the influence of specific training data from a model without requiring full retraining.<n>We propose REMIND, a novel evaluation method aiming to detect the subtle remaining influence of unlearned data.<n>We show that unlearned data yield flatter, less steep loss landscapes, while retained or unrelated data exhibit sharper, more volatile patterns.
  arXiv  Detail & Related papers  (2025-11-06T09:58:19Z)
• Unlearning Isn't Deletion: Investigating Reversibility of Machine Unlearning in LLMs [38.837810490068556]
  Unlearning in large language models (LLMs) aims to remove specified data, but its efficacy is typically assessed with task-level metrics like accuracy and perplexity.<n>We demonstrate that models can appear to forget while their original behavior is easily restored through minimal fine-tuning.<n>This phenomenon of emphreversibility suggests that information is merely suppressed, not genuinely erased.
  arXiv  Detail & Related papers  (2025-05-22T16:02:10Z)
• Erasing Without Remembering: Implicit Knowledge Forgetting in Large Language Models [81.62767292169225]
  We investigate knowledge forgetting in large language models with a focus on its generalisation.<n>We propose PerMU, a novel probability perturbation-based unlearning paradigm.<n>Experiments are conducted on a diverse range of datasets, including TOFU, Harry Potter, ZsRE, WMDP, and MUSE.
  arXiv  Detail & Related papers  (2025-02-27T11:03:33Z)
• TruVRF: Towards Triple-Granularity Verification on Machine Unlearning [4.3667223256713745]
  TruVRF is a non-invasive unlearning verification framework operating at class-, volume-, and sample-level granularities. It includes three Unlearning-Metrics designed to detect different types of dishonest servers: Neglecting, Lazy, and Deceiving. We show TruVRF's robust performance, with over 90% accuracy for Metrics I and III, and a 4.8% to 8.2% inference deviation for Metric II.
  arXiv  Detail & Related papers  (2024-08-12T11:29:54Z)
• Rethinking Classifier Re-Training in Long-Tailed Recognition: A Simple Logits Retargeting Approach [102.0769560460338]
  We develop a simple logits approach (LORT) without the requirement of prior knowledge of the number of samples per class. Our method achieves state-of-the-art performance on various imbalanced datasets, including CIFAR100-LT, ImageNet-LT, and iNaturalist 2018.
  arXiv  Detail & Related papers  (2024-03-01T03:27:08Z)
• Unsupervised Continual Anomaly Detection with Contrastively-learned Prompt [80.43623986759691]
  We introduce a novel Unsupervised Continual Anomaly Detection framework called UCAD. The framework equips the UAD with continual learning capability through contrastively-learned prompts. We conduct comprehensive experiments and set the benchmark on unsupervised continual anomaly detection and segmentation.
  arXiv  Detail & Related papers  (2024-01-02T03:37:11Z)
• Dual Learning for Large Vocabulary On-Device ASR [64.10124092250128]
  Dual learning is a paradigm for semi-supervised machine learning that seeks to leverage unsupervised data by solving two opposite tasks at once. We provide an analysis of an on-device-sized streaming conformer trained on the entirety of Librispeech, showing relative WER improvements of 10.7%/5.2% without an LM and 11.7%/16.4% with an LM.
  arXiv  Detail & Related papers  (2023-01-11T06:32:28Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.