Boosting Alignment for Post-Unlearning Text-to-Image Generative Models

• URL: http://arxiv.org/abs/2412.07808v1
• Date: Mon, 09 Dec 2024 21:36:10 GMT
• Title: Boosting Alignment for Post-Unlearning Text-to-Image Generative Models
• Authors: Myeongseob Ko, Henry Li, Zhun Wang, Jonathan Patsenker, Jiachen T. Wang, Qinbin Li, Ming Jin, Dawn Song, Ruoxi Jia,
• Abstract summary: Large-scale generative models have shown impressive image-generation capabilities, propelled by massive data. This often inadvertently leads to the generation of harmful or inappropriate content and raises copyright concerns. We propose a framework that seeks an optimal model update at each unlearning iteration, ensuring monotonic improvement on both objectives.
• Score: 55.82190434534429
• License:
• Abstract: Large-scale generative models have shown impressive image-generation capabilities, propelled by massive data. However, this often inadvertently leads to the generation of harmful or inappropriate content and raises copyright concerns. Driven by these concerns, machine unlearning has become crucial to effectively purge undesirable knowledge from models. While existing literature has studied various unlearning techniques, these often suffer from either poor unlearning quality or degradation in text-image alignment after unlearning, due to the competitive nature of these objectives. To address these challenges, we propose a framework that seeks an optimal model update at each unlearning iteration, ensuring monotonic improvement on both objectives. We further derive the characterization of such an update. In addition, we design procedures to strategically diversify the unlearning and remaining datasets to boost performance improvement. Our evaluation demonstrates that our method effectively removes target classes from recent diffusion-based generative models and concepts from stable diffusion models while maintaining close alignment with the models' original trained states, thus outperforming state-of-the-art baselines. Our code will be made available at \url{https://github.com/reds-lab/Restricted_gradient_diversity_unlearning.git}.

Related papers

• Efficient Fine-Tuning and Concept Suppression for Pruned Diffusion Models [93.76814568163353]
  We propose a novel bilevel optimization framework for pruned diffusion models. This framework consolidates the fine-tuning and unlearning processes into a unified phase. It is compatible with various pruning and concept unlearning methods.
  arXiv  Detail & Related papers  (2024-12-19T19:13:18Z)
• RADIOv2.5: Improved Baselines for Agglomerative Vision Foundation Models [60.596005921295806]
  Agglomerative models have emerged as a powerful approach to training vision foundation models. We identify critical challenges including resolution mode shifts, teacher imbalance, idiosyncratic teacher artifacts, and an excessive number of output tokens. We propose several novel solutions: multi-resolution training, mosaic augmentation, and improved balancing of teacher loss functions.
  arXiv  Detail & Related papers  (2024-12-10T17:06:41Z)
• Reward Incremental Learning in Text-to-Image Generation [26.64026346266299]
  We present Reward Incremental Distillation (RID), a method that mitigates forgetting with minimal computational overhead. The experimental results demonstrate the efficacy of RID in achieving consistent, high-quality gradient generation in RIL scenarios.
  arXiv  Detail & Related papers  (2024-11-26T10:54:33Z)
• Model Integrity when Unlearning with T2I Diffusion Models [11.321968363411145]
  We propose approximate Machine Unlearning algorithms to reduce the generation of specific types of images, characterized by samples from a forget distribution'' We then propose unlearning algorithms that demonstrate superior effectiveness in preserving model integrity compared to existing baselines.
  arXiv  Detail & Related papers  (2024-11-04T13:15:28Z)
• Adversarial Robustification via Text-to-Image Diffusion Models [56.37291240867549]
  Adrial robustness has been conventionally believed as a challenging property to encode for neural networks. We develop a scalable and model-agnostic solution to achieve adversarial robustness without using any data.
  arXiv  Detail & Related papers  (2024-07-26T10:49:14Z)
• Reinforcing Pre-trained Models Using Counterfactual Images [54.26310919385808]
  This paper proposes a novel framework to reinforce classification models using language-guided generated counterfactual images. We identify model weaknesses by testing the model using the counterfactual image dataset. We employ the counterfactual images as an augmented dataset to fine-tune and reinforce the classification model.
  arXiv  Detail & Related papers  (2024-06-19T08:07:14Z)
• ObjBlur: A Curriculum Learning Approach With Progressive Object-Level Blurring for Improved Layout-to-Image Generation [7.645341879105626]
  We present Blur, a novel curriculum learning approach to improve layout-to-image generation models. Our method is based on progressive object-level blurring, which effectively stabilizes training and enhances the quality of generated images.
  arXiv  Detail & Related papers  (2024-04-11T08:50:12Z)
• Active Generation for Image Classification [45.93535669217115]
  We propose to address the efficiency of image generation by focusing on the specific needs and characteristics of the model. With a central tenet of active learning, our method, named ActGen, takes a training-aware approach to image generation.
  arXiv  Detail & Related papers  (2024-03-11T08:45:31Z)
• Conditional Generation from Unconditional Diffusion Models using Denoiser Representations [94.04631421741986]
  We propose adapting pre-trained unconditional diffusion models to new conditions using the learned internal representations of the denoiser network. We show that augmenting the Tiny ImageNet training set with synthetic images generated by our approach improves the classification accuracy of ResNet baselines by up to 8%.
  arXiv  Detail & Related papers  (2023-06-02T20:09:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.