ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models

• URL: http://arxiv.org/abs/2605.18879v2
• Date: Wed, 20 May 2026 08:28:44 GMT
• Title: ZeroUnlearn: Few-Shot Knowledge Unlearning in Large Language Models
• Abstract summary: Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations.<n>Existing machine unlearning methods rely on retraining or aggressive fine-tuning.<n>We propose ZeroUnlearn, a few-shot unlearning framework that overwrites sensitive inputs by mapping them to a neutral target state.
• Score: 36.71886205422977
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large language models inevitably retain sensitive information, defined as inputs that may induce harmful generations, due to training on massive web corpora, raising concerns for privacy and safety. Existing machine unlearning methods primarily rely on retraining or aggressive fine-tuning, which are either computationally expensive or prone to degrading related knowledge and overall model utility. In this work, we reformulate machine unlearning as a precise knowledge re-mapping problem via model editing. We propose ZeroUnlearn, a few-shot unlearning framework. It overwrites sensitive inputs by mapping them to a neutral target state and removing their original representations. ZeroUnlearn enforces representational orthogonality through a multiplicative parameter update with a closed-form solution, enabling efficient and targeted unlearning. We further extend ZeroUnlearn to a gradient-based variant for multi-sample unlearning. Experiments demonstrate that our approach outperforms existing baselines while preserving general model utility. Our code is available at the github: https://github.com/XMUDeepLIT/ZeroUnlearn.

Related papers

• Fast Unlearning at Scale via Margin Self-Correction [52.46927918952516]
  Language-model unlearning updates a trained model to behave as if it had not seen selected training examples.<n>MASC achieves a competitive forget-retain trade-off at a fraction of the computational cost of existing baselines.
  arXiv  Detail & Related papers  (2026-06-01T21:49:54Z)
• Rethinking Machine Unlearning: Models Designed to Forget via Key Deletion [18.166109788922153]
  We propose unlearning by design, a novel paradigm in which models are directly trained to support forgetting.<n>We instantiate this idea with Machine UNlearning via KEY deletion (MUNKEY), a memory augmented transformer that decouples instance-specific memorization from model weights.<n>Our results establish that unlearning by design enables fast, deployment-oriented unlearning while preserving predictive performance.
  arXiv  Detail & Related papers  (2026-03-16T09:33:29Z)
• Model Collapse Is Not a Bug but a Feature in Machine Unlearning for LLMs [54.167494079321465]
  Current unlearning methods for LLMs optimize on the private information they seek to remove by incorporating it into their fine-tuning data.<n>We propose a novel unlearning method-Partial Model Collapse (PMC), which does not require unlearning targets in the unlearning objective.
  arXiv  Detail & Related papers  (2025-07-06T03:08:49Z)
• Align-then-Unlearn: Embedding Alignment for LLM Unlearning [41.94295877935867]
  Unlearning seeks to selectively remove specific data from trained models, such as personal information or copyrighted content.<n>We propose Align-then-Unlearn, a novel framework that performs unlearning in the semantic embedding space.
  arXiv  Detail & Related papers  (2025-06-16T07:48:01Z)
• Learn while Unlearn: An Iterative Unlearning Framework for Generative Language Models [52.40798352740857]
  We introduce the Iterative Contrastive Unlearning (ICU) framework, which consists of three core components.<n>A Knowledge Unlearning Induction module targets specific knowledge for removal using an unlearning loss.<n>A Contrastive Learning Enhancement module preserves the model's expressive capabilities against the pure unlearning goal.<n>An Iterative Unlearning Refinement module dynamically adjusts the unlearning process through ongoing evaluation and updates.
  arXiv  Detail & Related papers  (2024-07-25T07:09:35Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• Can Bad Teaching Induce Forgetting? Unlearning in Deep Networks using an Incompetent Teacher [6.884272840652062]
  We propose a novel machine unlearning method by exploring the utility of competent and incompetent teachers in a student-teacher framework to induce forgetfulness. The knowledge from the competent and incompetent teachers is selectively transferred to the student to obtain a model that doesn't contain any information about the forget data. We introduce the zero forgetting (ZRF) metric to evaluate any unlearning method.
  arXiv  Detail & Related papers  (2022-05-17T05:13:17Z)
• Machine Unlearning of Features and Labels [72.81914952849334]
  We propose first scenarios for unlearning and labels in machine learning models. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
  arXiv  Detail & Related papers  (2021-08-26T04:42:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.