SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion

• URL: http://arxiv.org/abs/2605.07482v1
• Date: Fri, 08 May 2026 09:25:18 GMT
• Title: SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion
• Abstract summary: SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion) is a retain-set-free unlearning method built on a key insight.<n>High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence.<n> SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.
• Score: 39.17638540496959
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning for large language models (LLMs) aims to selectively remove memorized content such as private data, copyrighted text, or hazardous knowledge, without costly full retraining. Most existing methods require a retain set of curated examples to prevent catastrophic degradation of general model utility, creating an extra data dependency that complicates deployment. We propose SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion), a retain-set-free unlearning method built on a key insight: not all tokens within a forget set instance carry memorized information equally. High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence. SHRED operates in two stages. (1) Selection: We perform a forward pass on a forget set instance, collect per-token autoregressive probabilities, and select the bottom (lowest probability, highest Shannon information) as forget positions; the remaining positions are retained as benign anchors. (2) Training: We construct modified KL targets that demote the memorized token's logit at forget positions while preserving the original distribution at benign positions. The model is then trained via a single top KL self-distillation objective that simultaneously drives forgetting and utility preservation. We evaluate SHRED across four standard unlearning benchmarks and demonstrate that it establishes a new Pareto-optimal trade-off between forget efficacy and model utility, outperforming retain-set-dependent methods. Our analysis shows that SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.

Related papers

• Confess What You Know: Forget-Set Misalignment with Model Knowledge in LLM Unlearning [9.140696856930457]
  We term this gap forget-set misalignment and identify two cases.<n>In Under Unlearning, the forget set omits memorized information and leakage persists.<n>In Out-of-Knowledge Unlearning, the algorithm is driven to "forget" knowledge the model never learned, perturbing parameters and degrading utility.<n>We propose a data-blind framework that constructs model-aligned forget sets by eliciting and formalizing the model's memorized knowledge.
  arXiv  Detail & Related papers  (2026-09-01T02:47:43Z)
• PURGE: Projected Unlearning via Retain-Guided Erasure [5.069332646113575]
  PURGE is a machine unlearning algorithm built on a simple but an under-exploited observation: continual learning (CL) and machine unlearning (MU)<n> CL tries to learn new tasks without forgetting old ones; MU tries to erase specific data without hurting retained performance representing the same underlying tension in opposite directions.<n> PURGE consistently keeps accuracy above 96% while achieving MIA AUROC close to 0.5 (the ideal), outperforming gradient ascent, KL-uniform, and several published baselines on the privacy-utility frontier.
  arXiv  Detail & Related papers  (2026-06-02T15:53:01Z)
• Before Forgetting, Learn to Remember: Revisiting Foundational Learning Failures in LVLM Unlearning Benchmarks [17.78493115982036]
  Large Vision-Language Models (LVLMs) pose privacy risks by unintentionally memorizing sensitive personal information.<n>We introduce ReMem, a Reliable Multi-hop and Multi-image Memorization Benchmark.<n>We show that ReMem provides a rigorous and trustworthy framework for diagnosing both learning and unlearning behaviors in LVLMs.
  arXiv  Detail & Related papers  (2026-05-05T13:42:31Z)
• Rethinking Machine Unlearning: Models Designed to Forget via Key Deletion [18.166109788922153]
  We propose unlearning by design, a novel paradigm in which models are directly trained to support forgetting.<n>We instantiate this idea with Machine UNlearning via KEY deletion (MUNKEY), a memory augmented transformer that decouples instance-specific memorization from model weights.<n>Our results establish that unlearning by design enables fast, deployment-oriented unlearning while preserving predictive performance.
  arXiv  Detail & Related papers  (2026-03-16T09:33:29Z)
• Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearning [71.96329385684395]
  We propose model extrapolation as an alternative to gradient ascent (GA)<n>Counterfactual as it might sound, a forget model can be obtained via extrapolation from the memorization model to the reference model.<n>Our model extrapolation is simple and efficient to implement, and it can also effectively converge throughout training to achieve improved unlearning performance.
  arXiv  Detail & Related papers  (2026-02-06T07:11:27Z)
• LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data [69.5099112089508]
  Large language models (LLMs) exhibit remarkable generative capabilities but raise ethical and security concerns by memorizing sensitive data.<n>This work presents the first study of unlearning under perturbed or low-fidelity forget data, referred to as noisy forget sets.<n>We find that unlearning remains surprisingly robust to perturbations, provided that core semantic signals are preserved.
  arXiv  Detail & Related papers  (2025-10-10T05:10:49Z)
• Pre-Forgettable Models: Prompt Learning as a Native Mechanism for Unlearning [9.512928441517811]
  Foundation models have transformed multimedia analysis by enabling robust and transferable representations across diverse modalities and tasks.<n>Traditional unlearning approaches, including retraining, activation editing, or distillation, are often expensive, fragile, and ill-suited for real-time or continuously evolving systems.<n>We introduce a prompt-based learning framework that unifies knowledge acquisition and removal within a single training phase.
  arXiv  Detail & Related papers  (2025-09-05T13:28:04Z)
• GUARD: Generation-time LLM Unlearning via Adaptive Restriction and Detection [36.38245533018162]
  Large Language Models (LLMs) have demonstrated strong capabilities in memorizing vast amounts of knowledge across diverse domains.<n>Existing unlearning efforts typically fine-tune the model with resources such as forget data, retain data, and a calibration model.<n>We propose Generation-time Unlearning via Adaptive Restriction and Detection (GUARD), a framework that enables dynamic unlearning during LLM generation.
  arXiv  Detail & Related papers  (2025-05-19T16:26:58Z)
• Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers [71.70205894168039]
  We consider instance-wise unlearning, of which the goal is to delete information on a set of instances from a pre-trained model. We propose two methods that reduce forgetting on the remaining data: 1) utilizing adversarial examples to overcome forgetting at the representation-level and 2) leveraging weight importance metrics to pinpoint network parameters guilty of propagating unwanted information.
  arXiv  Detail & Related papers  (2023-01-27T07:53:50Z)
• Machine Unlearning of Features and Labels [72.81914952849334]
  We propose first scenarios for unlearning and labels in machine learning models. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
  arXiv  Detail & Related papers  (2021-08-26T04:42:24Z)
• Automatic Recall Machines: Internal Replay, Continual Learning and the Brain [104.38824285741248]
  Replay in neural networks involves training on sequential data with memorized samples, which counteracts forgetting of previous behavior caused by non-stationarity. We present a method where these auxiliary samples are generated on the fly, given only the model that is being trained for the assessed objective. Instead the implicit memory of learned samples within the assessed model itself is exploited.
  arXiv  Detail & Related papers  (2020-06-22T15:07:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.