Representation Unlearning: Forgetting through Information Compression

• URL: http://arxiv.org/abs/2601.21564v1
• Date: Thu, 29 Jan 2026 11:28:02 GMT
• Title: Representation Unlearning: Forgetting through Information Compression
• Authors: Antonio Almudévar, Alfonso Ortega,
• Abstract summary: We introduce Representation Unlearning, a framework that performs unlearning directly in the model's representation space.<n>We show that Representation Unlearning achieves more reliable forgetting, better utility retention, and greater computational efficiency than parameter-centric baselines.
• Score: 3.9189279162842854
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning seeks to remove the influence of specific training data from a model, a need driven by privacy regulations and robustness concerns. Existing approaches typically modify model parameters, but such updates can be unstable, computationally costly, and limited by local approximations. We introduce Representation Unlearning, a framework that performs unlearning directly in the model's representation space. Instead of modifying model parameters, we learn a transformation over representations that imposes an information bottleneck: maximizing mutual information with retained data while suppressing information about data to be forgotten. We derive variational surrogates that make this objective tractable and show how they can be instantiated in two practical regimes: when both retain and forget data are available, and in a zero-shot setting where only forget data can be accessed. Experiments across several benchmarks demonstrate that Representation Unlearning achieves more reliable forgetting, better utility retention, and greater computational efficiency than parameter-centric baselines.

Related papers

• Forgetting-MarI: LLM Unlearning via Marginal Information Regularization [6.979586479353831]
  Existing unlearning methods often degrade model performance by removing more information than necessary when attempting to ''forget'' specific data.<n>We introduce Forgetting-MarI, an LLM unlearning framework that provably removes only the additional (marginal) information contributed by the data to be unlearned.<n>By penalizing marginal information, our method yields an explicit upper bound on the unlearn dataset's residual influence in the trained models, providing provable undetectability.
  arXiv  Detail & Related papers  (2025-11-14T22:48:39Z)
• Reveal and Release: Iterative LLM Unlearning with Self-generated Data [5.932877449308903]
  We propose a Reveal-and-Release'' method to unlearn with self-generated data.<n>We make incremental adjustments to the model's weight space with parameter-efficient modules trained on the forget data.
  arXiv  Detail & Related papers  (2025-09-18T05:07:27Z)
• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• Encapsulating Knowledge in One Prompt [56.31088116526825]
  KiOP encapsulates knowledge from various models into a solitary prompt without altering the original models or requiring access to the training data. From a practicality standpoint, this paradigm proves the effectiveness of Visual Prompt in data inaccessible contexts. Experiments across various datasets and models demonstrate the efficacy of the proposed KiOP knowledge transfer paradigm.
  arXiv  Detail & Related papers  (2024-07-16T16:35:23Z)
• The Frontier of Data Erasure: Machine Unlearning for Large Language Models [56.26002631481726]
  Large Language Models (LLMs) are foundational to AI advancements. LLMs pose risks by potentially memorizing and disseminating sensitive, biased, or copyrighted information. Machine unlearning emerges as a cutting-edge solution to mitigate these concerns.
  arXiv  Detail & Related papers  (2024-03-23T09:26:15Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• Learning to Unlearn: Instance-wise Unlearning for Pre-trained Classifiers [71.70205894168039]
  We consider instance-wise unlearning, of which the goal is to delete information on a set of instances from a pre-trained model. We propose two methods that reduce forgetting on the remaining data: 1) utilizing adversarial examples to overcome forgetting at the representation-level and 2) leveraging weight importance metrics to pinpoint network parameters guilty of propagating unwanted information.
  arXiv  Detail & Related papers  (2023-01-27T07:53:50Z)
• Disentangling representations in Restricted Boltzmann Machines without adversaries [0.0]
  We propose a simple, effective way of disentangling representations without any need to train adversarial discriminators. We show how our framework allows for computing the cost, in terms of log-likelihood of the data, associated to the disentanglement of their representations.
  arXiv  Detail & Related papers  (2022-06-23T10:24:20Z)
• Machine Unlearning of Features and Labels [72.81914952849334]
  We propose first scenarios for unlearning and labels in machine learning models. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
  arXiv  Detail & Related papers  (2021-08-26T04:42:24Z)
• Model-agnostic and Scalable Counterfactual Explanations via Reinforcement Learning [0.5729426778193398]
  We propose a deep reinforcement learning approach that transforms the optimization procedure into an end-to-end learnable process. Our experiments on real-world data show that our method is model-agnostic, relying only on feedback from model predictions.
  arXiv  Detail & Related papers  (2021-06-04T16:54:36Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.