Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models

• URL: http://arxiv.org/abs/2405.12523v2
• Date: Wed, 29 May 2024 05:27:38 GMT
• Title: Single Image Unlearning: Efficient Machine Unlearning in Multimodal Large Language Models
• Authors: Jiaqi Li, Qianshan Wei, Chuanyi Zhang, Guilin Qi, Miaozeng Du, Yongrui Chen, Sheng Bi,
• Abstract summary: We propose an efficient method, Single Image Unlearning (SIU), to unlearn the visual recognition of a concept by fine-tuning a single associated image for few steps. Experimental results on MMUBench show that SIU completely surpasses the performance of existing methods.
• Score: 13.08771725554285
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning empowers individuals with the `right to be forgotten' by removing their private or sensitive information encoded in machine learning models. However, it remains uncertain whether MU can be effectively applied to Multimodal Large Language Models (MLLMs), particularly in scenarios of forgetting the leaked visual data of concepts. To overcome the challenge, we propose an efficient method, Single Image Unlearning (SIU), to unlearn the visual recognition of a concept by fine-tuning a single associated image for few steps. SIU consists of two key aspects: (i) Constructing Multifaceted fine-tuning data. We introduce four targets, based on which we construct fine-tuning data for the concepts to be forgotten; (ii) Jointly training loss. To synchronously forget the visual recognition of concepts and preserve the utility of MLLMs, we fine-tune MLLMs through a novel Dual Masked KL-divergence Loss combined with Cross Entropy loss. Alongside our method, we establish MMUBench, a new benchmark for MU in MLLMs and introduce a collection of metrics for its evaluation. Experimental results on MMUBench show that SIU completely surpasses the performance of existing methods. Furthermore, we surprisingly find that SIU can avoid invasive membership inference attacks and jailbreak attacks. To the best of our knowledge, we are the first to explore MU in MLLMs. We will release the code and benchmark in the near future.

Related papers

• SAFEERASER: Enhancing Safety in Multimodal Large Language Models through Multimodal Machine Unlearning [19.9759585536617]
  We propose SAFEERASER, a safety unlearning benchmark for Multimodal Large Language Models (MLLMs) We comprehensively evaluate unlearning methods from two perspectives: forget quality and model utility. Experimental results demonstrate that combining PD Loss with existing unlearning methods can effectively prevent over-forgetting.
  arXiv  Detail & Related papers  (2025-02-18T04:09:46Z)
• MMUNLEARNER: Reformulating Multimodal Machine Unlearning in the Era of Multimodal Large Language Models [19.36626553745877]
  We reformulate the task of multimodal Machine Unlearning (MU) in the era of Multimodal Large Language Models (MLLMs) We develop a novel geometry-constrained descent gradient method MMUnlearner. It updates the weights of MLLMs with a weight saliency map jointly restricted by the remaining concepts and textual knowledge during unlearning.
  arXiv  Detail & Related papers  (2025-02-16T09:23:50Z)
• Looking Beyond Text: Reducing Language bias in Large Vision-Language Models via Multimodal Dual-Attention and Soft-Image Guidance [67.26434607115392]
  Large vision-language models (LVLMs) have achieved impressive results in various vision-language tasks. LVLMs suffer from hallucinations caused by language bias, leading to diminished focus on images and ineffective visual comprehension. We propose LACING to address the language bias of LVLMs with muLtimodal duAl-attention meChanIsm (MDA) aNd soft-image Guidance (IFG)
  arXiv  Detail & Related papers  (2024-11-21T16:33:30Z)
• A Closer Look at Machine Unlearning for Large Language Models [46.245404272612795]
  Large language models (LLMs) may memorize sensitive or copyrighted content, raising privacy and legal concerns. We discuss several issues in machine unlearning for LLMs and provide our insights on possible approaches.
  arXiv  Detail & Related papers  (2024-10-10T16:56:05Z)
• NoteLLM-2: Multimodal Large Representation Models for Recommendation [71.87790090964734]
  Large Language Models (LLMs) have demonstrated exceptional proficiency in text understanding and embedding tasks. Their potential in multimodal representation, particularly for item-to-item (I2I) recommendations, remains underexplored. We propose an end-to-end fine-tuning method that customizes the integration of any existing LLMs and vision encoders for efficient multimodal representation.
  arXiv  Detail & Related papers  (2024-05-27T03:24:01Z)
• MLLMReID: Multimodal Large Language Model-based Person Re-identification [14.68436005777866]
  Multimodal large language models (MLLM) have achieved satisfactory results in many tasks. This paper will investigate how to adapt them for the task of ReID.
  arXiv  Detail & Related papers  (2024-01-24T03:07:26Z)
• Machine Vision Therapy: Multimodal Large Language Models Can Enhance Visual Robustness via Denoising In-Context Learning [67.0609518552321]
  We propose to conduct Machine Vision Therapy which aims to rectify the noisy predictions from vision models. By fine-tuning with the denoised labels, the learning model performance can be boosted in an unsupervised manner.
  arXiv  Detail & Related papers  (2023-12-05T07:29:14Z)
• MOCA: Self-supervised Representation Learning by Predicting Masked Online Codebook Assignments [72.6405488990753]
  Self-supervised learning can be used for mitigating the greedy needs of Vision Transformer networks. We propose a single-stage and standalone method, MOCA, which unifies both desired properties. We achieve new state-of-the-art results on low-shot settings and strong experimental results in various evaluation protocols.
  arXiv  Detail & Related papers  (2023-07-18T15:46:20Z)
• MME: A Comprehensive Evaluation Benchmark for Multimodal Large Language Models [73.86954509967416]
  Multimodal Large Language Model (MLLM) relies on the powerful LLM to perform multimodal tasks. This paper presents the first comprehensive MLLM Evaluation benchmark MME. It measures both perception and cognition abilities on a total of 14 subtasks.
  arXiv  Detail & Related papers  (2023-06-23T09:22:36Z)
• Multi-Modal Few-Shot Temporal Action Detection [157.96194484236483]
  Few-shot (FS) and zero-shot (ZS) learning are two different approaches for scaling temporal action detection to new classes. We introduce a new multi-modality few-shot (MMFS) TAD problem, which can be considered as a marriage of FS-TAD and ZS-TAD.
  arXiv  Detail & Related papers  (2022-11-27T18:13:05Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.