Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model

• URL: http://arxiv.org/abs/2504.01521v1
• Date: Wed, 02 Apr 2025 09:07:55 GMT
• Title: Domain Guidance: A Simple Transfer Approach for a Pre-trained Diffusion Model
• Authors: Jincheng Zhong, Xiangcheng Zhang, Jianmin Wang, Mingsheng Long,
• Abstract summary: Domain Guidance is a transfer approach that leverages pre-trained knowledge to guide the sampling process toward the target domain.<n>We demonstrate its substantial effectiveness across various transfer benchmarks, achieving over a 19.6% improvement in FID and a 23.4% improvement in FD$_textDINOv2$ compared to standard fine-tuning.
• Score: 62.11981915549919
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Recent advancements in diffusion models have revolutionized generative modeling. However, the impressive and vivid outputs they produce often come at the cost of significant model scaling and increased computational demands. Consequently, building personalized diffusion models based on off-the-shelf models has emerged as an appealing alternative. In this paper, we introduce a novel perspective on conditional generation for transferring a pre-trained model. From this viewpoint, we propose *Domain Guidance*, a straightforward transfer approach that leverages pre-trained knowledge to guide the sampling process toward the target domain. Domain Guidance shares a formulation similar to advanced classifier-free guidance, facilitating better domain alignment and higher-quality generations. We provide both empirical and theoretical analyses of the mechanisms behind Domain Guidance. Our experimental results demonstrate its substantial effectiveness across various transfer benchmarks, achieving over a 19.6% improvement in FID and a 23.4% improvement in FD$_\text{DINOv2}$ compared to standard fine-tuning. Notably, existing fine-tuned models can seamlessly integrate Domain Guidance to leverage these benefits, without additional training.

Related papers

• Variational Control for Guidance in Diffusion Models [19.51536406897083]
  We introduce Diffusion Trajectory Matching (DTM) that enables guiding pretrained diffusion trajectories to satisfy a terminal cost.<n>DTM unifies a broad class of guidance methods and enables novel instantiations.<n>For instance, in ImageNet non-linear deblurring, our model achieves an FID score of 34.31, significantly improving over the best pretrained-method baseline (FID 78.07)
  arXiv  Detail & Related papers  (2025-02-06T00:24:39Z)
• Domain Generalization Guided by Large-Scale Pre-Trained Priors [24.74398777539288]
  Domain generalization (DG) aims to train a model from limited source domains, allowing it to generalize to unknown target domains. We introduce Fine-Tune with Large-scale pre-trained Priors (FT-LP) FT-LP incorporates the pre-trained model as a prior into the DG fine-tuning process, ensuring that the model refers to its pre-trained model at each optimization step.
  arXiv  Detail & Related papers  (2024-06-09T03:32:32Z)
• Bridging Model-Based Optimization and Generative Modeling via Conservative Fine-Tuning of Diffusion Models [54.132297393662654]
  We introduce a hybrid method that fine-tunes cutting-edge diffusion models by optimizing reward models through RL. We demonstrate the capability of our approach to outperform the best designs in offline data, leveraging the extrapolation capabilities of reward models.
  arXiv  Detail & Related papers  (2024-05-30T03:57:29Z)
• Transfer Learning for Diffusion Models [43.10840361752551]
  Diffusion models consistently produce high-quality synthetic samples. They can be impractical in real-world applications due to high collection costs or associated risks. This paper introduces the Transfer Guided Diffusion Process (TGDP), a novel approach distinct from conventional finetuning and regularization methods.
  arXiv  Detail & Related papers  (2024-05-27T06:48:58Z)
• Self-Play Fine-Tuning of Diffusion Models for Text-to-Image Generation [59.184980778643464]
  Fine-tuning Diffusion Models remains an underexplored frontier in generative artificial intelligence (GenAI) In this paper, we introduce an innovative technique called self-play fine-tuning for diffusion models (SPIN-Diffusion) Our approach offers an alternative to conventional supervised fine-tuning and RL strategies, significantly improving both model performance and alignment.
  arXiv  Detail & Related papers  (2024-02-15T18:59:18Z)
• Image is All You Need to Empower Large-scale Diffusion Models for In-Domain Generation [7.1629002695210024]
  In-domain generation aims to perform a variety of tasks within a specific domain, such as unconditional generation, text-to-image, image editing, 3D generation, and more.<n>Early research typically required training specialized generators for each unique task and domain, often relying on fully-labeled data.<n>Motivated by the powerful generative capabilities and broad applications of diffusion models, we are driven to explore leveraging label-free data to empower these models for in-domain generation.
  arXiv  Detail & Related papers  (2023-12-13T14:59:49Z)
• Guided Flows for Generative Modeling and Decision Making [55.42634941614435]
  We show that Guided Flows significantly improves the sample quality in conditional image generation and zero-shot text synthesis-to-speech. Notably, we are first to apply flow models for plan generation in the offline reinforcement learning setting ax speedup in compared to diffusion models.
  arXiv  Detail & Related papers  (2023-11-22T15:07:59Z)
• Elucidating The Design Space of Classifier-Guided Diffusion Generation [17.704873767509557]
  We show that it is possible to achieve significant performance improvements over existing guidance schemes by leveraging off-the-shelf classifiers in a training-free fashion. Our proposed approach has great potential and can be readily scaled up to text-to-image generation tasks.
  arXiv  Detail & Related papers  (2023-10-17T14:34:58Z)
• Improving Transferability of Adversarial Examples via Bayesian Attacks [84.90830931076901]
  We introduce a novel extension by incorporating the Bayesian formulation into the model input as well, enabling the joint diversification of both the model input and model parameters. Our method achieves a new state-of-the-art on transfer-based attacks, improving the average success rate on ImageNet and CIFAR-10 by 19.14% and 2.08%, respectively.
  arXiv  Detail & Related papers  (2023-07-21T03:43:07Z)
• Diff-Instruct: A Universal Approach for Transferring Knowledge From Pre-trained Diffusion Models [77.83923746319498]
  We propose a framework called Diff-Instruct to instruct the training of arbitrary generative models. We show that Diff-Instruct results in state-of-the-art single-step diffusion-based models. Experiments on refining GAN models show that the Diff-Instruct can consistently improve the pre-trained generators of GAN models.
  arXiv  Detail & Related papers  (2023-05-29T04:22:57Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.