Inference-Time Machine Unlearning via Gated Activation Redirection

• URL: http://arxiv.org/abs/2605.12765v2
• Date: Mon, 18 May 2026 16:57:42 GMT
• Title: Inference-Time Machine Unlearning via Gated Activation Redirection
• Abstract summary: We introduce Inference-Time Unlearning via Gated Activation Redirection (GUARD-IT)<n>GUARD-IT unlearns via input-dependent activation steering at inference time, leaving model weights untouched.<n>Experiments on TOFU and MUSE show that GUARD-IT matches or exceeds 12 gradient-based baselines across three model scales.
• Score: 1.1699531043716684
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Large Language Models memorize vast amounts of training data, raising concerns regarding privacy, copyright infringement, and safety. Machine unlearning seeks to remove the influence of a targeted forget set while preserving model performance, ideally approximating a model retrained from scratch without the forget set. Existing approaches aim to achieve this by updating model parameters via gradient-based methods. However, these updates are computationally expensive, lead to irreversible weight changes, and degrade when the model is quantized for deployment. A recent alternative to changing model weights is activation engineering, where activations are changed during inference to steer model behavior. Despite circumventing weight editing, naive activation steering introduces its own failure modes, as a single global steering vector applies the same intervention to every input, leading to unintended changes in model behavior. We introduce Inference-Time Unlearning via Gated Activation Redirection (GUARD-IT), a training- and gradient-free method that unlearns via input-dependent activation steering at inference time. The resulting intervention is applied as a norm-preserving rotation in the residual stream, leaving model weights untouched. Experiments on TOFU and MUSE show that GUARD-IT matches or exceeds 12 gradient-based baselines across three model scales, while being the only method to simultaneously preserve utility, suppress memorization, and avoid catastrophic collapse across all settings. GUARD-IT further supports continual unlearning without retraining, and remains effective under quantization, a scenario in which parameter-editing methods degrade.

Related papers

• Test-Time Unlearning via Sparse Autoencoder [55.4290003355764]
  We propose ARIA, a test-time unlearning method that leaves model weights intact and gates access to unwanted knowledge only when generation enters a forget-related state.<n>ARIA improves the forget-retain trade-off over weight-based baselines across both a thinking model and an instruction model.
  arXiv  Detail & Related papers  (2026-09-14T18:58:09Z)
• Not Every Subject Should Stay: Machine Unlearning for Noisy Engagement Recognition [53.005382593686356]
  Engagement recognition datasets are typically subject-indexed and often contain noisy, subjective supervision.<n>We study this setting through subject-level machine unlearning as a post-hoc sanitization mechanism for engagement recognition.
  arXiv  Detail & Related papers  (2026-05-06T10:03:06Z)
• RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair [1.7118181664522618]
  Large language models (LLMs) inherently absorb harmful knowledge, misinformation, and personal data during pretraining on large-scale web corpora.<n>We introduce Interactive Machine Unlearning (IMU), a new paradigm in which users can instruct LLMs to forget targeted knowledge through natural language at inference time.<n>We develop RePAIR, a prompt-aware model repair framework comprising (i) a watchdog model for unlearning intent detection, (ii) a surgeon model for generating repair procedures, and (iii) a patient model whose parameters are updated autonomously.
  arXiv  Detail & Related papers  (2026-04-14T14:44:45Z)
• From Gradients to Riccati Geometry: Kalman World Models for Single-Pass Learning [0.0]
  Backpropagation dominates modern machine learning, yet it is not the only principled method for optimizing dynamical systems.<n>We propose Kalman World Models (KWM), a class of learned state-space models trained via Bayesian filtering.<n>We extend this framework to transformer-based large language models (LLMs), where internal activations are treated as latent dynamical states corrected.
  arXiv  Detail & Related papers  (2026-03-13T00:30:30Z)
• Is Gradient Ascent Really Necessary? Memorize to Forget for Machine Unlearning [71.96329385684395]
  We propose model extrapolation as an alternative to gradient ascent (GA)<n>Counterfactual as it might sound, a forget model can be obtained via extrapolation from the memorization model to the reference model.<n>Our model extrapolation is simple and efficient to implement, and it can also effectively converge throughout training to achieve improved unlearning performance.
  arXiv  Detail & Related papers  (2026-02-06T07:11:27Z)
• UniErase: Towards Balanced and Precise Unlearning in Language Models [69.04923022755547]
  Large language models (LLMs) require iterative updates to address the outdated information problem.<n>UniErase is a novel unlearning framework that demonstrates precision and balanced performances between knowledge unlearning and ability retaining.
  arXiv  Detail & Related papers  (2025-05-21T15:53:28Z)
• Machine Unlearning on Pre-trained Models by Residual Feature Alignment Using LoRA [15.542668474378633]
  We propose a novel and efficient machine unlearning method on pre-trained models. We leverage LoRA to decompose the model's intermediate features into pre-trained features and residual features. The method aims to learn the zero residuals on the retained set and shifted residuals on the unlearning set.
  arXiv  Detail & Related papers  (2024-11-13T08:56:35Z)
• Training Language Models to Self-Correct via Reinforcement Learning [98.35197671595343]
  Self-correction has been found to be largely ineffective in modern large language models (LLMs) We develop a multi-turn online reinforcement learning approach, SCoRe, that significantly improves an LLM's self-correction ability using entirely self-generated data. We find that SCoRe achieves state-of-the-art self-correction performance, improving the base models' self-correction by 15.6% and 9.1% respectively on MATH and HumanEval.
  arXiv  Detail & Related papers  (2024-09-19T17:16:21Z)
• Learn to Unlearn for Deep Neural Networks: Minimizing Unlearning Interference with Gradient Projection [56.292071534857946]
  Recent data-privacy laws have sparked interest in machine unlearning. Challenge is to discard information about the forget'' data without altering knowledge about remaining dataset. We adopt a projected-gradient based learning method, named as Projected-Gradient Unlearning (PGU) We provide empirically evidence to demonstrate that our unlearning method can produce models that behave similar to models retrained from scratch across various metrics even when the training dataset is no longer accessible.
  arXiv  Detail & Related papers  (2023-12-07T07:17:24Z)
• Gradient Surgery for One-shot Unlearning on Generative Model [0.989293617504294]
  We introduce a simple yet effective approach to remove a data influence on the deep generative model. Inspired by works in multi-task learning, we propose to manipulate gradients to regularize the interplay of influence among samples.
  arXiv  Detail & Related papers  (2023-07-10T13:29:23Z)
• Machine Unlearning of Features and Labels [72.81914952849334]
  We propose first scenarios for unlearning and labels in machine learning models. Our approach builds on the concept of influence functions and realizes unlearning through closed-form updates of model parameters.
  arXiv  Detail & Related papers  (2021-08-26T04:42:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.