Test-Time Unlearning via Sparse Autoencoder

• URL: http://arxiv.org/abs/2609.16229v1
• Date: Mon, 14 Sep 2026 18:58:09 GMT
• Title: Test-Time Unlearning via Sparse Autoencoder
• Abstract summary: We propose ARIA, a test-time unlearning method that leaves model weights intact and gates access to unwanted knowledge only when generation enters a forget-related state.<n>ARIA improves the forget-retain trade-off over weight-based baselines across both a thinking model and an instruction model.
• Score: 55.4290003355764
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning aims to remove specific knowledge from a trained large language model (LLM) without retraining from scratch. Existing methods modify model weights via gradient ascent and its advances. While effective on certain benchmarks, these weight-based approaches exhibit a sharp forget-utility trade-off, where stronger forgetting of target knowledge can degrade model utility, and unlearned knowledge may reappear under post-unlearning fine-tuning or prompt attacks. We propose ARIA (autoencoder-gated inference-time unlearning), a test-time unlearning method that leaves model weights intact and gates access to unwanted knowledge only when generation enters a forget-related state. ARIA uses sparse autoencoder (SAE) latents to train a lightweight linear detector, then applies an interpretable intervention on triggered states with negligible test-time overhead. Empirical evaluations on TOFU, R-TOFU, and WMDP show that ARIA improves the forget-retain trade-off over weight-based baselines across both a thinking model (DeepSeek-R1-Distilled-Qwen-1.5B) and an instruction model (Gemma-3-1B-it), e.g., reducing WMDP-cyber forget-set accuracy significantly while keeping MMLU within 1% of the pre-unlearning model. We further introduce three post-unlearning adversarial attacks targeting weight-space and decoding-space recovery, and find that ARIA remains robust under all three, with forgetting changing by less than 1% under attack. A feature-level case study leveraging the interpretability of ARIA suggests that some retain degradation may reflect response styles underlying the unlearning data rather than leakage of the targeted knowledge itself, highlighting a potential source of bias in unlearning task construction.

Related papers

• LACUNA: A Testbed for Evaluating Localization Precision for LLM Unlearning [30.67510673327925]
  We introduce LACUNA: the first unlearning testbed with ground-truth parameter-level localization.<n>We show that LACUNA achieves strong erasure and robustness to resurfacing attacks, highlighting the importance of precise unlearning.
  arXiv  Detail & Related papers  (2026-07-02T17:59:52Z)
• Distinguishable Deletion: Unifying Knowledge Erasure and Refusal for Large Language Model Unlearning [58.725080160369494]
  Distinguishable Deletion ($mathrmD2$) is a paradigm that restricts the response distribution in the latent representation rather than specific tokens.<n>We introduce an energy index that quantifies the presence of knowledge and the separation between unlearned and retained content.<n>Experiments demonstrate that EUA significantly outperforms previous methods, indicating the superiority of $mathrmD2$.
  arXiv  Detail & Related papers  (2026-05-16T03:15:35Z)
• RePAIR: Interactive Machine Unlearning through Prompt-Aware Model Repair [1.7118181664522618]
  Large language models (LLMs) inherently absorb harmful knowledge, misinformation, and personal data during pretraining on large-scale web corpora.<n>We introduce Interactive Machine Unlearning (IMU), a new paradigm in which users can instruct LLMs to forget targeted knowledge through natural language at inference time.<n>We develop RePAIR, a prompt-aware model repair framework comprising (i) a watchdog model for unlearning intent detection, (ii) a surgeon model for generating repair procedures, and (iii) a patient model whose parameters are updated autonomously.
  arXiv  Detail & Related papers  (2026-04-14T14:44:45Z)
• Scalable and Robust LLM Unlearning by Correcting Responses with Retrieved Exclusions [49.55618517046225]
  Language models trained on web-scale corpora risk memorizing and exposing sensitive information.<n>We propose Corrective Unlearning with Retrieved Exclusions (CURE), a novel unlearning framework.<n>CURE verifies model outputs for leakage and revises them into safe responses.
  arXiv  Detail & Related papers  (2025-09-30T09:07:45Z)
• Efficient Machine Unlearning via Influence Approximation [75.31015485113993]
  Influence-based unlearning has emerged as a prominent approach to estimate the impact of individual training samples on model parameters without retraining.<n>This paper establishes a theoretical link between memorizing (incremental learning) and forgetting (unlearning)<n>We introduce the Influence Approximation Unlearning algorithm for efficient machine unlearning from the incremental perspective.
  arXiv  Detail & Related papers  (2025-07-31T05:34:27Z)
• OPC: One-Point-Contraction Unlearning Toward Deep Feature Forgetting [2.6815971241599126]
  Machine unlearning seeks to remove the influence of particular data or class from trained models to meet privacy, legal, or ethical requirements.<n>Existing unlearning methods tend to forget shallowly: phenomenon of an unlearned model pretend to forget by adjusting only the model response.<n>We propose a novel general-purpose unlearning algorithm: One-Point-Contraction (OPC)
  arXiv  Detail & Related papers  (2025-07-10T13:34:02Z)
• Robust Anti-Backdoor Instruction Tuning in LVLMs [53.766434746801366]
  We introduce a lightweight, certified-agnostic defense framework for large visual language models (LVLMs)<n>Our framework finetunes only adapter modules and text embedding layers under instruction tuning.<n>Experiments against seven attacks on Flickr30k and MSCOCO demonstrate that ours reduces their attack success rate to nearly zero.
  arXiv  Detail & Related papers  (2025-06-04T01:23:35Z)
• UniErase: Towards Balanced and Precise Unlearning in Language Models [69.04923022755547]
  Large language models (LLMs) require iterative updates to address the outdated information problem.<n>UniErase is a novel unlearning framework that demonstrates precision and balanced performances between knowledge unlearning and ability retaining.
  arXiv  Detail & Related papers  (2025-05-21T15:53:28Z)
• Towards Robust and Parameter-Efficient Knowledge Unlearning for LLMs [25.91643745340183]
  Large Language Models (LLMs) have demonstrated strong reasoning and memorization capabilities via pretraining on massive textual corpora.<n>This poses risk of privacy and copyright violations, highlighting the need for efficient machine unlearning methods.<n>We propose Low-rank Knowledge Unlearning (LoKU), a novel framework that enables robust and efficient unlearning for LLMs.
  arXiv  Detail & Related papers  (2024-08-13T04:18:32Z)
• Unrolling SGD: Understanding Factors Influencing Machine Unlearning [17.6607904333012]
  Machine unlearning is the process through which a deployed machine learning model forgets about one of its training data points. We first taxonomize approaches and metrics of approximate unlearning. We identify verification error, i.e., the L2 difference between the weights of an approximately unlearned and a naively retrained model.
  arXiv  Detail & Related papers  (2021-09-27T23:46:59Z)
• AGKD-BML: Defense Against Adversarial Attack by Attention Guided Knowledge Distillation and Bi-directional Metric Learning [61.8003954296545]
  We propose a novel adversarial training-based model by Attention Guided Knowledge Distillation and Bi-directional Metric Learning (AGKD-BML) Our proposed AGKD-BML model consistently outperforms the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2021-08-13T01:25:04Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.