Automating Evaluation of Diffusion Model Unlearning with (Vision-) Language Model World Knowledge

• URL: http://arxiv.org/abs/2507.07137v1
• Date: Wed, 09 Jul 2025 00:51:09 GMT
• Title: Automating Evaluation of Diffusion Model Unlearning with (Vision-) Language Model World Knowledge
• Authors: Eric Yeats, Darryl Hannan, Henry Kvinge, Timothy Doster, Scott Mahan,
• Abstract summary: Machine unlearning (MU) is a cost-effective method to cleanse undesired information (generated concepts, biases, or patterns) from foundational diffusion models.<n>We introduce autoeval-dmun, an automated tool which leverages (vision-) language models to thoroughly assess unlearning in diffusion models.<n>Given a target concept, autoeval-dmun extracts structured, relevant world knowledge from the language model to identify nearby concepts which are likely damaged by unlearning.
• Score: 6.4411440750013735
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Machine unlearning (MU) is a promising cost-effective method to cleanse undesired information (generated concepts, biases, or patterns) from foundational diffusion models. While MU is orders of magnitude less costly than retraining a diffusion model without the undesired information, it can be challenging and labor-intensive to prove that the information has been fully removed from the model. Moreover, MU can damage diffusion model performance on surrounding concepts that one would like to retain, making it unclear if the diffusion model is still fit for deployment. We introduce autoeval-dmun, an automated tool which leverages (vision-) language models to thoroughly assess unlearning in diffusion models. Given a target concept, autoeval-dmun extracts structured, relevant world knowledge from the language model to identify nearby concepts which are likely damaged by unlearning and to circumvent unlearning with adversarial prompts. We use our automated tool to evaluate popular diffusion model unlearning methods, revealing that language models (1) impose semantic orderings of nearby concepts which correlate well with unlearning damage and (2) effectively circumvent unlearning with synthetic adversarial prompts.

Related papers

• DiffusionTrend: A Minimalist Approach to Virtual Fashion Try-On [103.89972383310715]
  DiffusionTrend harnesses latent information rich in prior information to capture the nuances of garment details.<n>It delivers a visually compelling try-on experience, underscoring the potential of training-free diffusion model.
  arXiv  Detail & Related papers  (2024-12-19T02:24:35Z)
• Diffusion-VLA: Generalizable and Interpretable Robot Foundation Model via Self-Generated Reasoning [9.923268972395107]
  DiffusionVLA is a framework that seamlessly combines the autoregression model with the diffusion model for learning visuomotor policy.<n>To enhance policy learning through self-reasoning, we introduce a novel reasoning injection module.<n>We conduct extensive experiments using multiple real robots to validate the effectiveness of DiffusionVLA.
  arXiv  Detail & Related papers  (2024-12-04T13:11:38Z)
• Diffusion Imitation from Observation [4.205946699819021]
  adversarial imitation learning approaches learn a generator agent policy to produce state transitions that are indistinguishable to a discriminator. Motivated by the recent success of diffusion models in generative modeling, we propose to integrate a diffusion model into the adversarial imitation learning from observation framework.
  arXiv  Detail & Related papers  (2024-10-07T18:49:55Z)
• Score Forgetting Distillation: A Swift, Data-Free Method for Machine Unlearning in Diffusion Models [63.43422118066493]
  Machine unlearning (MU) is a crucial foundation for developing safe, secure, and trustworthy GenAI models.<n>Traditional MU methods often rely on stringent assumptions and require access to real data.<n>This paper introduces Score Forgetting Distillation (SFD), an innovative MU approach that promotes the forgetting of undesirable information in diffusion models.
  arXiv  Detail & Related papers  (2024-09-17T14:12:50Z)
• Unlearning or Concealment? A Critical Analysis and Evaluation Metrics for Unlearning in Diffusion Models [7.9993879763024065]
  This paper presents a theoretical and empirical examination of five commonly used techniques for unlearning in diffusion models.<n>We introduce two new evaluation metrics: Concept Retrieval Score (textbfCRS) and Concept Confidence Score (textbfCCS)
  arXiv  Detail & Related papers  (2024-09-09T14:38:31Z)
• Diffusion of Thoughts: Chain-of-Thought Reasoning in Diffusion Language Models [100.53662473219806]
  Diffusion-of-Thought (DoT) is a novel approach that integrates diffusion models with Chain-of-Thought.<n>DoT allows reasoning steps to diffuse over time through a diffusion language model.<n>Our results demonstrate the effectiveness of DoT in multi-digit multiplication, logic, and grade school math problems.
  arXiv  Detail & Related papers  (2024-02-12T16:23:28Z)
• Training Diffusion Models with Reinforcement Learning [82.29328477109826]
  Diffusion models are trained with an approximation to the log-likelihood objective. In this paper, we investigate reinforcement learning methods for directly optimizing diffusion models for downstream objectives. We describe how posing denoising as a multi-step decision-making problem enables a class of policy gradient algorithms.
  arXiv  Detail & Related papers  (2023-05-22T17:57:41Z)
• DiffusionBERT: Improving Generative Masked Language Models with Diffusion Models [81.84866217721361]
  DiffusionBERT is a new generative masked language model based on discrete diffusion models. We propose a new noise schedule for the forward diffusion process that controls the degree of noise added at each step. Experiments on unconditional text generation demonstrate that DiffusionBERT achieves significant improvement over existing diffusion models for text.
  arXiv  Detail & Related papers  (2022-11-28T03:25:49Z)
• Beyond Trivial Counterfactual Explanations with Diverse Valuable Explanations [64.85696493596821]
  In computer vision applications, generative counterfactual methods indicate how to perturb a model's input to change its prediction. We propose a counterfactual method that learns a perturbation in a disentangled latent space that is constrained using a diversity-enforcing loss. Our model improves the success rate of producing high-quality valuable explanations when compared to previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2021-03-18T12:57:34Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.