Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearning

• URL: http://arxiv.org/abs/2410.05664v1
• Date: Tue, 8 Oct 2024 03:30:39 GMT
• Title: Holistic Unlearning Benchmark: A Multi-Faceted Evaluation for Text-to-Image Diffusion Model Unlearning
• Authors: Saemi Moon, Minjong Lee, Sangdon Park, Dongwoo Kim,
• Abstract summary: We release our comprehensive evaluation framework with the source codes and artifacts. Our investigation reveals that every method has side effects or limitations, especially in more complex and realistic situations.
• Score: 8.831339626121848
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: As text-to-image diffusion models become advanced enough for commercial applications, there is also increasing concern about their potential for malicious and harmful use. Model unlearning has been proposed to mitigate the concerns by removing undesired and potentially harmful information from the pre-trained model. So far, the success of unlearning is mainly measured by whether the unlearned model can generate a target concept while maintaining image quality. However, unlearning is typically tested under limited scenarios, and the side effects of unlearning have barely been studied in the current literature. In this work, we thoroughly analyze unlearning under various scenarios with five key aspects. Our investigation reveals that every method has side effects or limitations, especially in more complex and realistic situations. By releasing our comprehensive evaluation framework with the source codes and artifacts, we hope to inspire further research in this area, leading to more reliable and effective unlearning methods.

Related papers

• On the Fairness, Diversity and Reliability of Text-to-Image Generative Models [49.60774626839712]
  multimodal generative models have sparked critical discussions on their fairness, reliability, and potential for misuse. We propose an evaluation framework designed to assess model reliability through their responses to perturbations in the embedding space. Our method lays the groundwork for detecting unreliable, bias-injected models and retrieval of bias provenance.
  arXiv  Detail & Related papers  (2024-11-21T09:46:55Z)
• Towards Understanding the Feasibility of Machine Unlearning [14.177012256360635]
  We present a set of novel metrics for quantifying the difficulty of unlearning. Specifically, we propose several metrics to assess the conditions necessary for a successful unlearning operation. We also present a ranking mechanism to identify the most challenging samples to unlearn.
  arXiv  Detail & Related papers  (2024-10-03T23:41:42Z)
• Towards Robust Knowledge Unlearning: An Adversarial Framework for Assessing and Improving Unlearning Robustness in Large Language Models [19.015202590038996]
  We design Dynamic Unlearning Attack (DUA), a dynamic and automated framework to attack unlearned models. We propose Latent Adrial Unlearning (LAU), a universal framework that effectively enhances the robustness of the unlearned process. We demonstrate that LAU improves unlearning effectiveness by over $53.5%$, cause only less than a $11.6%$ reduction in neighboring knowledge, and have almost no impact on the model's general capabilities.
  arXiv  Detail & Related papers  (2024-08-20T09:36:04Z)
• Unlearning with Control: Assessing Real-world Utility for Large Language Model Unlearning [97.2995389188179]
  Recent research has begun to approach large language models (LLMs) unlearning via gradient ascent (GA) Despite their simplicity and efficiency, we suggest that GA-based methods face the propensity towards excessive unlearning. We propose several controlling methods that can regulate the extent of excessive unlearning.
  arXiv  Detail & Related papers  (2024-06-13T14:41:00Z)
• UMat: Uncertainty-Aware Single Image High Resolution Material Capture [2.416160525187799]
  We propose a learning-based method to recover normals, specularity, and roughness from a single diffuse image of a material. Our method is the first one to deal with the problem of modeling uncertainty in material digitization.
  arXiv  Detail & Related papers  (2023-05-25T17:59:04Z)
• Harnessing the Power of Text-image Contrastive Models for Automatic Detection of Online Misinformation [50.46219766161111]
  We develop a self-learning model to explore the constrastive learning in the domain of misinformation identification. Our model shows the superior performance of non-matched image-text pair detection when the training data is insufficient.
  arXiv  Detail & Related papers  (2023-04-19T02:53:59Z)
• Learnware: Small Models Do Big [69.88234743773113]
  The prevailing big model paradigm, which has achieved impressive results in natural language processing and computer vision applications, has not yet addressed those issues, whereas becoming a serious source of carbon emissions. This article offers an overview of the learnware paradigm, which attempts to enable users not need to build machine learning models from scratch, with the hope of reusing small models to do things even beyond their original purposes.
  arXiv  Detail & Related papers  (2022-10-07T15:55:52Z)
• What Makes Good Contrastive Learning on Small-Scale Wearable-based Tasks? [59.51457877578138]
  We study contrastive learning on the wearable-based activity recognition task. This paper presents an open-source PyTorch library textttCL-HAR, which can serve as a practical tool for researchers.
  arXiv  Detail & Related papers  (2022-02-12T06:10:15Z)
• Plausible Counterfactuals: Auditing Deep Learning Classifiers with Realistic Adversarial Examples [84.8370546614042]
  Black-box nature of Deep Learning models has posed unanswered questions about what they learn from data. Generative Adversarial Network (GAN) and multi-objectives are used to furnish a plausible attack to the audited model. Its utility is showcased within a human face classification task, unveiling the enormous potential of the proposed framework.
  arXiv  Detail & Related papers  (2020-03-25T11:08:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.