Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning

• URL: http://arxiv.org/abs/2607.18615v1
• Date: Tue, 21 Jul 2026 01:35:33 GMT
• Title: Stochastic Meta-Unlearning: Bridging Language Backbone and Multimodal Unlearning
• Abstract summary: We show that VLM-level feedback can make language-backbone unlearning more reliable and more transferable for vision-language models.<n>We propose a bilevel framework that uses VLM-level feedback to learn unlearning-ready initialization.
• Score: 58.99072471887724
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning for vision-language models (VLMs) remains underexplored. Unlike language models, VLMs combine a language backbone with visual components, which makes unlearning more complex. There is a surprising phenomenon when moving from single-modality unlearning to VLM unlearning: a target forgotten by the standalone language backbone can still be recovered when image information is given to the full VLM. This shows that text-only feedback is not enough for reliable VLM unlearning. Motivated by this observation, we propose Stochastic Meta-Unlearning (SMU), a bilevel framework that uses VLM-level feedback to learn an unlearning-ready initialization. In the inner loop, SMU applies a few unlearning steps to the language backbone using text data. In the outer loop, SMU recomposes the updated backbone with the frozen VLM and evaluates forgetting and utility at the VLM level. This design makes the unlearning update aware of the final multimodal behavior, while still keeping the update local to the language backbone. Experiments on two VLMs, two multimodal meme datasets, and three baselines show that SMU achieves the best overall forget-retain trade-off. Compared with the strongest baseline for each metric, SMU reduces average Forget accuracy by 10.52 points and improves average Retain and Test accuracy by 20.10 and 17.01 points, respectively. More importantly, SMU also transfers to new forgetting targets and to different meta-test unlearning methods. These results suggest that VLM-level feedback can make language-backbone unlearning more reliable and more transferable for VLMs.

Related papers

• OFFSIDE: Benchmarking Unlearning Misinformation in Multimodal Large Language Models [42.313806202695176]
  We introduce OFFSIDE, a novel benchmark for evaluating misinformation unlearning in MLLMs based on football transfer rumors.<n>This dataset contains 15.68K records for 80 players, providing a comprehensive framework with four test sets to assess forgetting efficacy, generalization, utility, and robustness.<n>Offside supports advanced settings like selective unlearning and corrective relearning, and crucially, unimodal unlearning.
  arXiv  Detail & Related papers  (2025-10-26T05:05:30Z)
• Continual Learning for VLMs: A Survey and Taxonomy Beyond Forgetting [70.83781268763215]
  Vision-language models (VLMs) have achieved impressive performance across diverse multimodal tasks by leveraging large-scale pre-training.<n>VLMs face unique challenges such as cross-modal feature drift, parameter interference due to shared architectures, and zero-shot capability erosion.<n>This survey aims to serve as a comprehensive and diagnostic reference for researchers developing lifelong vision-language systems.
  arXiv  Detail & Related papers  (2025-08-06T09:03:10Z)
• Evaluating Cell Type Inference in Vision Language Models Under Varying Visual Context [0.16385815610837165]
  Vision-Language Models (VLMs) have rapidly advanced alongside Large Language Models (LLMs)<n>This study evaluates the capabilities of prominent generative VLMs, such as GPT-4.1 and Gemini 2.5 Pro, for histopathology image classification tasks.
  arXiv  Detail & Related papers  (2025-06-15T01:50:16Z)
• MMUnlearner: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models [19.36626553745877]
  We propose to reformulate the task of multimodal Machine Unlearning (MU) in the era of Multimodal Large Language Models (MLLMs)<n>We develop a novel geometry-constrained gradient ascent method MMUnlearner.<n>It updates the weights of MLLMs with a weight saliency map jointly restricted by the remaining concepts and textual knowledge during unlearning.
  arXiv  Detail & Related papers  (2025-02-16T09:23:50Z)
• Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance [67.26434607115392]
  Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. LVLMs suffer from hallucinations caused by language bias, leading to diminished focus on images and ineffective visual comprehension. We propose LACING to address the language bias of LVLMs with muLtimodal duAl-attention meChanIsm (MDA) aNd soft-image Guidance (IFG)
  arXiv  Detail & Related papers  (2024-11-21T16:33:30Z)
• Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models [16.886116549737956]
  We propose an efficient method, Single Image Unlearning (SIU), to unlearn the visual recognition of a concept by fine-tuning a single associated image for few steps.<n> Experimental results on MMUBench show that SIU completely surpasses the performance of existing methods.
  arXiv  Detail & Related papers  (2024-05-21T06:27:12Z)
• The Neglected Tails in Vision-Language Models [51.79913798808725]
  We show that vision-language models (VLMs) excel in zero-shot recognition but their performance varies greatly across different visual concepts. We propose REtrieval-Augmented Learning (REAL) to mitigate the imbalanced performance of zero-shot VLMs.
  arXiv  Detail & Related papers  (2024-01-23T01:25:00Z)
• VILA: On Pre-training for Visual Language Models [74.08039416548209]
  We study the design options for VLM pre-training through step-by-step controllable comparisons. We build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-12T18:58:18Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [111.51612340032052]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks.<n>This paper presents the first comprehensive MLLM Evaluation benchmark MME.<n>It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)
• LVLM-eHub: A Comprehensive Evaluation Benchmark for Large Vision-Language Models [55.304181390027274]
  This paper presents a comprehensive evaluation of publicly available large multimodal models by building a LVLM evaluation Hub (LVLM-eHub) Our LVLM-eHub consists of $8$ representative LVLMs such as InstructBLIP and MiniGPT-4, which are thoroughly evaluated by a quantitative capability evaluation and an online arena platform. The study reveals several innovative findings. First, instruction-tuned LVLM with massive in-domain data such as InstructBLIP heavily overfits many existing tasks, generalizing poorly in the open-world scenario.
  arXiv  Detail & Related papers  (2023-06-15T16:39:24Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.