DiT4SR: Taming Diffusion Transformer for Real-World Image Super-Resolution

• URL: http://arxiv.org/abs/2503.23580v1
• Date: Sun, 30 Mar 2025 20:27:22 GMT
• Title: DiT4SR: Taming Diffusion Transformer for Real-World Image Super-Resolution
• Authors: Zheng-Peng Duan, Jiawei Zhang, Xin Jin, Ziheng Zhang, Zheng Xiong, Dongqing Zou, Jimmy Ren, Chun-Le Guo, Chongyi Li,
• Abstract summary: DiT4SR is one of the pioneering works to tame the large-scale DiT model for Real-ISR.<n>Instead of directly injecting embeddings extracted from low-resolution (LR) images like ControlNet, we integrate the LR embeddings into the original attention mechanism of DiT.<n>The LR guidance is injected into the generated latent via a cross-stream convolution layer, compensating for DiT's limited ability to capture local information.
• Score: 42.26299332658843
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: Large-scale pre-trained diffusion models are becoming increasingly popular in solving the Real-World Image Super-Resolution (Real-ISR) problem because of their rich generative priors. The recent development of diffusion transformer (DiT) has witnessed overwhelming performance over the traditional UNet-based architecture in image generation, which also raises the question: Can we adopt the advanced DiT-based diffusion model for Real-ISR? To this end, we propose our DiT4SR, one of the pioneering works to tame the large-scale DiT model for Real-ISR. Instead of directly injecting embeddings extracted from low-resolution (LR) images like ControlNet, we integrate the LR embeddings into the original attention mechanism of DiT, allowing for the bidirectional flow of information between the LR latent and the generated latent. The sufficient interaction of these two streams allows the LR stream to evolve with the diffusion process, producing progressively refined guidance that better aligns with the generated latent at each diffusion step. Additionally, the LR guidance is injected into the generated latent via a cross-stream convolution layer, compensating for DiT's limited ability to capture local information. These simple but effective designs endow the DiT model with superior performance in Real-ISR, which is demonstrated by extensive experiments. Project Page: https://adam-duan.github.io/projects/dit4sr/.

Related papers

• Single-Step Latent Consistency Model for Remote Sensing Image Super-Resolution [7.920423405957888]
  We propose a novel single-step diffusion approach designed to enhance both efficiency and visual quality in RSISR tasks. The proposed LCMSR reduces the iterative steps of traditional diffusion models from 50-1000 or more to just a single step. Experimental results demonstrate that LCMSR effectively balances efficiency and performance, achieving inference times comparable to non-diffusion models.
  arXiv  Detail & Related papers  (2025-03-25T09:56:21Z)
• One Diffusion Step to Real-World Super-Resolution via Flow Trajectory Distillation [60.54811860967658]
  FluxSR is a novel one-step diffusion Real-ISR based on flow matching models.<n>First, we introduce Flow Trajectory Distillation (FTD) to distill a multi-step flow matching model into a one-step Real-ISR.<n>Second, to improve image realism and address high-frequency artifact issues in generated images, we propose TV-LPIPS as a perceptual loss.
  arXiv  Detail & Related papers  (2025-02-04T04:11:29Z)
• TSD-SR: One-Step Diffusion with Target Score Distillation for Real-World Image Super-Resolution [25.994093587158808]
  Pre-trained text-to-image diffusion models are increasingly applied to real-world image super-resolution (Real-ISR) tasks.<n>Given the iterative refinement nature of diffusion models, most existing approaches are computationally expensive.<n>We propose TSD-SR, a novel distillation framework specifically designed for real-world image super-resolution.
  arXiv  Detail & Related papers  (2024-11-27T12:01:08Z)
• ClearSR: Latent Low-Resolution Image Embeddings Help Diffusion-Based Real-World Super Resolution Models See Clearer [68.72454974431749]
  We present ClearSR, a new method that can better take advantage of latent low-resolution image (LR) embeddings for diffusion-based real-world image super-resolution (Real-ISR) Our model can achieve better performance across multiple metrics on several test sets and generate more consistent SR results with LR images than existing methods.
  arXiv  Detail & Related papers  (2024-10-18T08:35:57Z)
• Enhanced Super-Resolution Training via Mimicked Alignment for Real-World Scenes [51.92255321684027]
  We propose a novel plug-and-play module designed to mitigate misalignment issues by aligning LR inputs with HR images during training. Specifically, our approach involves mimicking a novel LR sample that aligns with HR while preserving the characteristics of the original LR samples. We comprehensively evaluate our method on synthetic and real-world datasets, demonstrating its effectiveness across a spectrum of SR models.
  arXiv  Detail & Related papers  (2024-10-07T18:18:54Z)
• Effective Diffusion Transformer Architecture for Image Super-Resolution [63.254644431016345]
  We design an effective diffusion transformer for image super-resolution (DiT-SR) In practice, DiT-SR leverages an overall U-shaped architecture, and adopts a uniform isotropic design for all the transformer blocks. We analyze the limitation of the widely used AdaLN, and present a frequency-adaptive time-step conditioning module.
  arXiv  Detail & Related papers  (2024-09-29T07:14:16Z)
• Binarized Diffusion Model for Image Super-Resolution [61.963833405167875]
  Binarization, an ultra-compression algorithm, offers the potential for effectively accelerating advanced diffusion models (DMs) Existing binarization methods result in significant performance degradation. We introduce a novel binarized diffusion model, BI-DiffSR, for image SR.
  arXiv  Detail & Related papers  (2024-06-09T10:30:25Z)
• LDM-ISP: Enhancing Neural ISP for Low Light with Latent Diffusion Models [54.93010869546011]
  We propose to leverage the pre-trained latent diffusion model to perform the neural ISP for enhancing extremely low-light images. Specifically, to tailor the pre-trained latent diffusion model to operate on the RAW domain, we train a set of lightweight taming modules. We observe different roles of UNet denoising and decoder reconstruction in the latent diffusion model, which inspires us to decompose the low-light image enhancement task into latent-space low-frequency content generation and decoding-phase high-frequency detail maintenance.
  arXiv  Detail & Related papers  (2023-12-02T04:31:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.