Single Trajectory Distillation for Accelerating Image and Video Style Transfer

• URL: http://arxiv.org/abs/2412.18945v1
• Date: Wed, 25 Dec 2024 16:40:23 GMT
• Title: Single Trajectory Distillation for Accelerating Image and Video Style Transfer
• Authors: Sijie Xu, Runqi Wang, Wei Zhu, Dejia Song, Nemo Chen, Xu Tang, Yao Hu,
• Abstract summary: Diffusion-based stylization methods typically denoise from a specific partial noise state for image-to-image and video-to-video tasks. We propose single trajectory distillation (STD) starting from a specific partial noise state. Our method surpasses existing acceleration models in terms of style similarity and aesthetic evaluations.
• Score: 22.304420035048942
• License:
• Abstract: Diffusion-based stylization methods typically denoise from a specific partial noise state for image-to-image and video-to-video tasks. This multi-step diffusion process is computationally expensive and hinders real-world application. A promising solution to speed up the process is to obtain few-step consistency models through trajectory distillation. However, current consistency models only force the initial-step alignment between the probability flow ODE (PF-ODE) trajectories of the student and the imperfect teacher models. This training strategy can not ensure the consistency of whole trajectories. To address this issue, we propose single trajectory distillation (STD) starting from a specific partial noise state. We introduce a trajectory bank to store the teacher model's trajectory states, mitigating the time cost during training. Besides, we use an asymmetric adversarial loss to enhance the style and quality of the generated images. Extensive experiments on image and video stylization demonstrate that our method surpasses existing acceleration models in terms of style similarity and aesthetic evaluations. Our code and results will be available on the project page: https://single-trajectory-distillation.github.io.

Related papers

• One Diffusion Step to Real-World Super-Resolution via Flow Trajectory Distillation [60.54811860967658]
  FluxSR is a novel one-step diffusion Real-ISR based on flow matching models. First, we introduce Flow Trajectory Distillation (FTD) to distill a multi-step flow matching model into a one-step Real-ISR. Second, to improve image realism and address high-frequency artifact issues in generated images, we propose TV-LPIPS as a perceptual loss.
  arXiv  Detail & Related papers  (2025-02-04T04:11:29Z)
• Accelerate High-Quality Diffusion Models with Inner Loop Feedback [50.00066451431194]
  Inner Loop Feedback (ILF) is a novel approach to accelerate diffusion models' inference. ILF trains a lightweight module to predict future features in the denoising process. ILF achieves strong performance for both class-to-image generation with diffusion transformer (DiT) and text-to-image generation with DiT-based PixArt-alpha and PixArt-sigma.
  arXiv  Detail & Related papers  (2025-01-22T18:59:58Z)
• OFTSR: One-Step Flow for Image Super-Resolution with Tunable Fidelity-Realism Trade-offs [20.652907645817713]
  OFTSR is a flow-based framework for one-step image super-resolution that can produce outputs with tunable levels of fidelity and realism. We demonstrate that OFTSR achieves state-of-the-art performance for one-step image super-resolution, while having the ability to flexibly tune the fidelity-realism trade-off.
  arXiv  Detail & Related papers  (2024-12-12T17:14:58Z)
• From Slow Bidirectional to Fast Autoregressive Video Diffusion Models [52.32078428442281]
  Current video diffusion models achieve impressive generation quality but struggle in interactive applications due to bidirectional attention dependencies. We address this limitation by adapting a pretrained bidirectional diffusion transformer to an autoregressive transformer that generates frames on-the-fly. Our model achieves a total score of 84.27 on the VBench-Long benchmark, surpassing all previous video generation models.
  arXiv  Detail & Related papers  (2024-12-10T18:59:50Z)
• Truncated Consistency Models [57.50243901368328]
  Training consistency models requires learning to map all intermediate points along PF ODE trajectories to their corresponding endpoints. We empirically find that this training paradigm limits the one-step generation performance of consistency models. We propose a new parameterization of the consistency function and a two-stage training procedure that prevents the truncated-time training from collapsing to a trivial solution.
  arXiv  Detail & Related papers  (2024-10-18T22:38:08Z)
• Distribution Backtracking Builds A Faster Convergence Trajectory for Diffusion Distillation [19.88187051373436]
  We propose Distribution Backtracking Distillation (DisBack) to speed up the sampling speed of diffusion models. DisBack achieves faster and better convergence than the existing distillation method, with FID score of 1.38 on ImageNet 64x64 dataset.
  arXiv  Detail & Related papers  (2024-08-28T17:58:17Z)
• SinSR: Diffusion-Based Image Super-Resolution in a Single Step [119.18813219518042]
  Super-resolution (SR) methods based on diffusion models exhibit promising results. But their practical application is hindered by the substantial number of required inference steps. We propose a simple yet effective method for achieving single-step SR generation, named SinSR.
  arXiv  Detail & Related papers  (2023-11-23T16:21:29Z)
• BOOT: Data-free Distillation of Denoising Diffusion Models with Bootstrapping [64.54271680071373]
  Diffusion models have demonstrated excellent potential for generating diverse images. Knowledge distillation has been recently proposed as a remedy that can reduce the number of inference steps to one or a few. We present a novel technique called BOOT, that overcomes limitations with an efficient data-free distillation algorithm.
  arXiv  Detail & Related papers  (2023-06-08T20:30:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.