Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2608.30649v1
• Date: Mon, 31 Aug 2026 11:51:05 GMT
• Title: Where Identity Lives: Localized, Retain-Free Identity Unlearning in Multimodal Large Language Models
• Abstract summary: Forgetting from the forget set alone damages the shared visual-language computation, harming perception.<n>We turn this into Pathway-Aware Visual-attribute Anchoring (PAVA), which updates to these layers and pairs a forget loss with a visual-attribute anchor that preserves image-grounded behavior by distilling the model's own pre-unlearning answers from the forget images alone.<n>On MLLMU-Bench and ReMem, PAVA gives the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.
• Score: 33.31500669671185
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Removing a specific individual's information from multimodal large language models (MLLMs) is often needed after deployment, but existing methods rely on a retain set, which is hardest to obtain at that point, and rebuilding it recreates the privacy exposure that unlearning aims to remove. Forgetting from the forget set alone instead damages the shared visual-language computation, harming perception. We cast retain-free unlearning as a localization problem: causal tracing, weight transplant, and Fisher overlap all point to early-to-mid decoder MLPs as the layers where identity information is stored and, unlike other module families, can be modified without substantially disrupting vision. We turn this into Pathway-Aware Visual-attribute Anchoring (PAVA), which confines updates to these layers and pairs a forget loss with a visual-attribute anchor that preserves image-grounded behavior by distilling the model's own pre-unlearning answers from the forget images alone. On MLLMU-Bench and ReMem, PAVA gives the strongest forget-retain trade-off among forget-set-only methods and remains competitive with retain-based baselines.

Related papers

• AIM: Anchor Identity Features, Then Match for Multimodal Large Language Model Unlearning [33.31500669671185]
  We study identity unlearning when retain images are unavailable at deletion time.<n>Our analysis shows that identity and visual-perception questions occupy distinct regions in fine-tuned hidden states.<n>We propose AIM, a two-stage method that anchors an identity-forgetting target with a universal visual prompt and then matches the vision encoder to that target.
  arXiv  Detail & Related papers  (2026-08-28T13:22:22Z)
• Toward Fine-Grained Forgetting:Attribute Unlearning for Multimodal Large Language Models [24.764929711765944]
  Machine unlearning seeks to remove designated knowledge without retraining from scratch while preserving general utility.<n>Existing privacy-oriented benchmarks primarily adopt profile-level deletion, whereas practical requests are often finer grained.<n>We introduce attribute-level MLLM unlearning as a finer-grained task and construct a benchmark spanning long-text, numeric, and short-text targets, multiple forget ratios, and diverse question types.<n>Our evaluation reveals that target and retained attributes share identity-specific and visual evidence, making selective forgetting susceptible to residual leakage or collateral degradation.
  arXiv  Detail & Related papers  (2026-08-02T05:23:44Z)
• SHRED: Retain-Set-Free Unlearning via Self-Distillation with Logit Demotion [39.17638540496959]
  SHRED (Self-distillation via High-surprisal-only Retain-set-free Entropy Demotion) is a retain-set-free unlearning method built on a key insight.<n>High-information tokens concentrate the model's memorized knowledge, while low-information tokens reflect general language competence.<n> SHRED is robust against relearning attacks and membership-inference attacks, and it maintains stable utility even after many sequential unlearning runs.
  arXiv  Detail & Related papers  (2026-05-08T09:25:18Z)
• MeGU: Machine-Guided Unlearning with Target Feature Disentanglement [73.49657372882082]
  We propose a novel framework that guides unlearning through concept-aware re-alignment.<n>MeGU enables controlled and selective forgetting, effectively mitigating both under-unlearning and over-unlearning.
  arXiv  Detail & Related papers  (2026-02-19T05:20:31Z)
• Agentic Learner with Grow-and-Refine Multimodal Semantic Memory [50.81667005063605]
  ViLoMem is a dual-stream memory framework that constructs compact, schema-based memory.<n>It encodes visual distraction patterns and logical reasoning errors, enabling MLLMs to learn from their successful and failed experiences.
  arXiv  Detail & Related papers  (2025-11-26T18:55:08Z)
• Towards Benign Memory Forgetting for Selective Multimodal Large Language Model Unlearning [49.274436951541425]
  Multimodal Large Language Models (MLLMs) achieve remarkable capabilities but can inadvertently memorize privacy-sensitive information.<n>Existing unlearning methods fail to achieve benign forgetting because they often degrade the model's general image understanding performance.<n>We propose the Sculpted Memory Forgetting Adapter (SMFA), which confines forgetting to targeted memory regions while preserving overall capabilities.
  arXiv  Detail & Related papers  (2025-11-25T11:22:45Z)
• AUVIC: Adversarial Unlearning of Visual Concepts for Multi-modal Large Language Models [63.05306474002547]
  Regulatory frameworks mandating the 'right to be forgotten' drive the need for machine unlearning.<n>We introduce AUVIC, a novel visual concept unlearning framework for MLLMs.<n>We show that AUVIC achieves state-of-the-art target forgetting rates while incurs minimal performance degradation on non-target concepts.
  arXiv  Detail & Related papers  (2025-11-14T13:35:32Z)
• LLM Unlearning on Noisy Forget Sets: A Study of Incomplete, Rewritten, and Watermarked Data [69.5099112089508]
  Large language models (LLMs) exhibit remarkable generative capabilities but raise ethical and security concerns by memorizing sensitive data.<n>This work presents the first study of unlearning under perturbed or low-fidelity forget data, referred to as noisy forget sets.<n>We find that unlearning remains surprisingly robust to perturbations, provided that core semantic signals are preserved.
  arXiv  Detail & Related papers  (2025-10-10T05:10:49Z)
• CLIPErase: Efficient Unlearning of Visual-Textual Associations in CLIP [57.49519639951552]
  We introduce CLIPErase, a novel approach that disentangles and selectively forgets both visual and textual associations.<n>Experiments on the CIFAR-100 and Flickr30K datasets demonstrate that CLIPErase effectively forgets designated associations in zero-shot tasks for multimodal samples.
  arXiv  Detail & Related papers  (2024-10-30T17:51:31Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.