Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization

• URL: http://arxiv.org/abs/2406.04314v3
• Date: Tue, 25 Mar 2025 17:06:27 GMT
• Title: Aesthetic Post-Training Diffusion Models from Generic Preferences with Step-by-step Preference Optimization
• Authors: Zhanhao Liang, Yuhui Yuan, Shuyang Gu, Bohan Chen, Tiankai Hang, Mingxi Cheng, Ji Li, Liang Zheng,
• Abstract summary: This paper introduces step-by-step preference optimization (SPO) to improve aesthetics economically.<n>SPO discards the propagation strategy and allows fine-grained image details to be assessed.<n>SPO converges much faster than DPO methods due to the use of more correct preference labels provided by the step-aware preference model.
• Score: 20.698818784349015
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generating visually appealing images is fundamental to modern text-to-image generation models. A potential solution to better aesthetics is direct preference optimization (DPO), which has been applied to diffusion models to improve general image quality including prompt alignment and aesthetics. Popular DPO methods propagate preference labels from clean image pairs to all the intermediate steps along the two generation trajectories. However, preference labels provided in existing datasets are blended with layout and aesthetic opinions, which would disagree with aesthetic preference. Even if aesthetic labels were provided (at substantial cost), it would be hard for the two-trajectory methods to capture nuanced visual differences at different steps. To improve aesthetics economically, this paper uses existing generic preference data and introduces step-by-step preference optimization (SPO) that discards the propagation strategy and allows fine-grained image details to be assessed. Specifically, at each denoising step, we 1) sample a pool of candidates by denoising from a shared noise latent, 2) use a step-aware preference model to find a suitable win-lose pair to supervise the diffusion model, and 3) randomly select one from the pool to initialize the next denoising step. This strategy ensures that diffusion models focus on the subtle, fine-grained visual differences instead of layout aspect. We find that aesthetics can be significantly enhanced by accumulating these improved minor differences. When fine-tuning Stable Diffusion v1.5 and SDXL, SPO yields significant improvements in aesthetics compared with existing DPO methods while not sacrificing image-text alignment compared with vanilla models. Moreover, SPO converges much faster than DPO methods due to the use of more correct preference labels provided by the step-aware preference model.

Related papers

• Fine-Tuning Diffusion Generative Models via Rich Preference Optimization [13.4078883626321]
  We introduce Rich Preference Optimization (RPO), a novel pipeline to improve the curation of preference pairs for fine-tuning text-to-image diffusion models. RPO generates detailed critiques of synthesized images to extract reliable and actionable image editing instructions. We demonstrate the effectiveness of our pipeline in fine-tuning state-of-the-art diffusion models.
  arXiv  Detail & Related papers  (2025-03-13T21:10:29Z)
• Dual Caption Preference Optimization for Diffusion Models [51.223275938663235]
  We propose Dual Caption Preference Optimization (DCPO), a novel approach that utilizes two distinct captions to mitigate irrelevant prompts. Our experiments show that DCPO significantly improves image quality and relevance to prompts, outperforming Stable Diffusion (SD) 2.1, SFT_Chosen, Diffusion-DPO, and MaPO across multiple metrics.
  arXiv  Detail & Related papers  (2025-02-09T20:34:43Z)
• Diffusion Model as a Noise-Aware Latent Reward Model for Step-Level Preference Optimization [46.888425016169144]
  Preference optimization for diffusion models aims to align them with human preferences for images. Previous methods typically leverage Vision-Language Models (VLMs) as pixel-level reward models to approximate human preferences. In this work, we demonstrate that diffusion models are inherently well-suited for step-level reward modeling in the latent space. We introduce Latent Preference Optimization (LPO), a method designed for step-level preference optimization directly in the latent space.
  arXiv  Detail & Related papers  (2025-02-03T04:51:28Z)
• Refining Alignment Framework for Diffusion Models with Intermediate-Step Preference Ranking [50.325021634589596]
  We propose a Tailored Optimization Preference (TailorPO) framework for aligning diffusion models with human preference. Our approach directly ranks intermediate noisy samples based on their step-wise reward, and effectively resolves the gradient direction issues. Experimental results demonstrate that our method significantly improves the model's ability to generate aesthetically pleasing and human-preferred images.
  arXiv  Detail & Related papers  (2025-02-01T16:08:43Z)
• Personalized Preference Fine-tuning of Diffusion Models [75.22218338096316]
  We introduce PPD, a multi-reward optimization objective that aligns diffusion models with personalized preferences. With PPD, a diffusion model learns the individual preferences of a population of users in a few-shot way. Our approach achieves an average win rate of 76% over Stable Cascade, generating images that more accurately reflect specific user preferences.
  arXiv  Detail & Related papers  (2025-01-11T22:38:41Z)
• PassionSR: Post-Training Quantization with Adaptive Scale in One-Step Diffusion based Image Super-Resolution [87.89013794655207]
  Diffusion-based image super-resolution (SR) models have shown superior performance at the cost of multiple denoising steps. We propose a novel post-training quantization approach with adaptive scale in one-step diffusion (OSD) image SR, PassionSR. Our PassionSR achieves significant advantages over recent leading low-bit quantization methods for image SR.
  arXiv  Detail & Related papers  (2024-11-26T04:49:42Z)
• Aligning Few-Step Diffusion Models with Dense Reward Difference Learning [81.85515625591884]
  Stepwise Diffusion Policy Optimization (SDPO) is an alignment method tailored for few-step diffusion models. SDPO incorporates dense reward feedback at every intermediate step to ensure consistent alignment across all denoising steps. SDPO consistently outperforms prior methods in reward-based alignment across diverse step configurations.
  arXiv  Detail & Related papers  (2024-11-18T16:57:41Z)
• Scalable Ranked Preference Optimization for Text-to-Image Generation [76.16285931871948]
  We investigate a scalable approach for collecting large-scale and fully synthetic datasets for DPO training. The preferences for paired images are generated using a pre-trained reward function, eliminating the need for involving humans in the annotation process. We introduce RankDPO to enhance DPO-based methods using the ranking feedback.
  arXiv  Detail & Related papers  (2024-10-23T16:42:56Z)
• MIA-DPO: Multi-Image Augmented Direct Preference Optimization For Large Vision-Language Models [85.30735602813093]
  Multi-Image Augmented Direct Preference Optimization (MIA-DPO) is a visual preference alignment approach that effectively handles multi-image inputs. MIA-DPO mitigates the scarcity of diverse multi-image training data by extending single-image data with unrelated images arranged in grid collages or pic-in-pic formats.
  arXiv  Detail & Related papers  (2024-10-23T07:56:48Z)
• Tuning Timestep-Distilled Diffusion Model Using Pairwise Sample Optimization [97.35427957922714]
  We present an algorithm named pairwise sample optimization (PSO), which enables the direct fine-tuning of an arbitrary timestep-distilled diffusion model. PSO introduces additional reference images sampled from the current time-step distilled model, and increases the relative likelihood margin between the training images and reference images. We show that PSO can directly adapt distilled models to human-preferred generation with both offline and online-generated pairwise preference image data.
  arXiv  Detail & Related papers  (2024-10-04T07:05:16Z)
• FIND: Fine-tuning Initial Noise Distribution with Policy Optimization for Diffusion Models [10.969811500333755]
  We introduce a Fine-tuning Initial Noise Distribution (FIND) framework with policy optimization. Our method achieves 10 times faster than the SOTA approach.
  arXiv  Detail & Related papers  (2024-07-28T10:07:55Z)
• Beta Sampling is All You Need: Efficient Image Generation Strategy for Diffusion Models using Stepwise Spectral Analysis [22.02829139522153]
  We propose an efficient time step sampling method based on an image spectral analysis of the diffusion process. Instead of the traditional uniform distribution-based time step sampling, we introduce a Beta distribution-like sampling technique. Our hypothesis is that certain steps exhibit significant changes in image content, while others contribute minimally.
  arXiv  Detail & Related papers  (2024-07-16T20:53:06Z)
• Towards More Accurate Diffusion Model Acceleration with A Timestep Aligner [84.97253871387028]
  A diffusion model, which is formulated to produce an image using thousands of denoising steps, usually suffers from a slow inference speed. We propose a timestep aligner that helps find a more accurate integral direction for a particular interval at the minimum cost. Experiments show that our plug-in design can be trained efficiently and boost the inference performance of various state-of-the-art acceleration methods.
  arXiv  Detail & Related papers  (2023-10-14T02:19:07Z)
• AutoDiffusion: Training-Free Optimization of Time Steps and Architectures for Automated Diffusion Model Acceleration [57.846038404893626]
  We propose to search the optimal time steps sequence and compressed model architecture in a unified framework to achieve effective image generation for diffusion models without any further training. Experimental results show that our method achieves excellent performance by using only a few time steps, e.g. 17.86 FID score on ImageNet 64 $times$ 64 with only four steps, compared to 138.66 with DDIM.
  arXiv  Detail & Related papers  (2023-09-19T08:57:24Z)
• Score Priors Guided Deep Variational Inference for Unsupervised Real-World Single Image Denoising [14.486289176696438]
  We propose a score priors-guided deep variational inference, namely ScoreDVI, for practical real-world denoising. We exploit a Non-$i.i.d$ Gaussian mixture model and variational noise posterior to model the real-world noise. Our method outperforms other single image-based real-world denoising methods and achieves comparable performance to dataset-based unsupervised methods.
  arXiv  Detail & Related papers  (2023-08-09T03:26:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.