Boosting Resolution Generalization of Diffusion Transformers with Randomized Positional Encodings

• URL: http://arxiv.org/abs/2503.18719v1
• Date: Mon, 24 Mar 2025 14:30:38 GMT
• Title: Boosting Resolution Generalization of Diffusion Transformers with Randomized Positional Encodings
• Authors: Cong Liu, Liang Hou, Mingwu Zheng, Xin Tao, Pengfei Wan, Di Zhang, Kun Gai,
• Abstract summary: We propose a novel two-dimensional randomized positional encodings framework that focuses on learning positional order of image patches instead of the specific distances between them.<n>Specifically, RPE-2D independently selects positions over a broader range along both the horizontal and vertical axes, ensuring that all position encodings are trained during the inference phase.<n>On the ImageNet dataset, our proposed RPE-2D achieves state-of-the-art resolution generalization performance, outperforming existing competitive methods when trained at a resolution of $256 times 256$ and inferred at $384 times 384$ and $512 times 512$
• Score: 27.46670809916215
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Resolution generalization in image generation tasks enables the production of higher-resolution images with lower training resolution overhead. However, a significant challenge in resolution generalization, particularly in the widely used Diffusion Transformers, lies in the mismatch between the positional encodings encountered during testing and those used during training. While existing methods have employed techniques such as interpolation, extrapolation, or their combinations, none have fully resolved this issue. In this paper, we propose a novel two-dimensional randomized positional encodings (RPE-2D) framework that focuses on learning positional order of image patches instead of the specific distances between them, enabling seamless high- and low-resolution image generation without requiring high- and low-resolution image training. Specifically, RPE-2D independently selects positions over a broader range along both the horizontal and vertical axes, ensuring that all position encodings are trained during the inference phase, thus improving resolution generalization. Additionally, we propose a random data augmentation technique to enhance the modeling of position order. To address the issue of image cropping caused by the augmentation, we introduce corresponding micro-conditioning to enable the model to perceive the specific cropping patterns. On the ImageNet dataset, our proposed RPE-2D achieves state-of-the-art resolution generalization performance, outperforming existing competitive methods when trained at a resolution of $256 \times 256$ and inferred at $384 \times 384$ and $512 \times 512$, as well as when scaling from $512 \times 512$ to $768 \times 768$ and $1024 \times 1024$. And it also exhibits outstanding capabilities in low-resolution image generation, multi-stage training acceleration and multi-resolution inheritance.

Related papers

• Zoomed In, Diffused Out: Towards Local Degradation-Aware Multi-Diffusion for Extreme Image Super-Resolution [8.69908615905782]
  Large-scale, pre-trained Text-to-Image (T2I) diffusion models have gained significant popularity in image generation tasks. Most existing T2I diffusion models are trained with a resolution limit of 512x512, making scaling beyond this resolution an unresolved but necessary challenge for image Super-Resolution (SR) We introduce a novel approach that, for the first time, enables these models to generate 2K, 4K, and even 8K images without any additional training.
  arXiv  Detail & Related papers  (2024-11-18T21:32:49Z)
• FlowDCN: Exploring DCN-like Architectures for Fast Image Generation with Arbitrary Resolution [33.07779971446476]
  We propose FlowDCN, a purely convolution-based generative model that can efficiently generate high-quality images at arbitrary resolutions. FlowDCN achieves the state-of-the-art 4.30 sFID on $256times256$ ImageNet Benchmark and comparable resolution extrapolation results. We believe FlowDCN offers a promising solution to scalable and flexible image synthesis.
  arXiv  Detail & Related papers  (2024-10-30T02:48:50Z)
• Hierarchical Patch Diffusion Models for High-Resolution Video Generation [50.42746357450949]
  We develop deep context fusion, which propagates context information from low-scale to high-scale patches in a hierarchical manner. We also propose adaptive computation, which allocates more network capacity and computation towards coarse image details. The resulting model sets a new state-of-the-art FVD score of 66.32 and Inception Score of 87.68 in class-conditional video generation.
  arXiv  Detail & Related papers  (2024-06-12T01:12:53Z)
• DiM: Diffusion Mamba for Efficient High-Resolution Image Synthesis [56.849285913695184]
  Diffusion Mamba (DiM) is a sequence model for efficient high-resolution image synthesis. DiM architecture achieves inference-time efficiency for high-resolution images. Experiments demonstrate the effectiveness and efficiency of our DiM.
  arXiv  Detail & Related papers  (2024-05-23T06:53:18Z)
• ScaleCrafter: Tuning-free Higher-Resolution Visual Generation with Diffusion Models [126.35334860896373]
  We investigate the capability of generating images from pre-trained diffusion models at much higher resolutions than the training image sizes. Existing works for higher-resolution generation, such as attention-based and joint-diffusion approaches, cannot well address these issues. We propose a simple yet effective re-dilation that can dynamically adjust the convolutional perception field during inference.
  arXiv  Detail & Related papers  (2023-10-11T17:52:39Z)
• CoordFill: Efficient High-Resolution Image Inpainting via Parameterized Coordinate Querying [52.91778151771145]
  In this paper, we try to break the limitations for the first time thanks to the recent development of continuous implicit representation. Experiments show that the proposed method achieves real-time performance on the 2048$times$2048 images using a single GTX 2080 Ti GPU.
  arXiv  Detail & Related papers  (2023-03-15T11:13:51Z)
• Raising The Limit Of Image Rescaling Using Auxiliary Encoding [7.9700865143145485]
  Recently, image rescaling models like IRN utilize the bidirectional nature of INN to push the performance limit of image upscaling. We propose auxiliary encoding modules to further push the limit of image rescaling performance.
  arXiv  Detail & Related papers  (2023-03-12T20:49:07Z)
• Single pixel imaging at high pixel resolutions [0.0]
  We show that image measurement at the full resolution of the DMD, which lasts only a fraction of a second, is possible for sparse images. We propose the sampling and reconstruction strategies that enable us to reconstruct sparse images at the resolution of $1024 times 768$ within the time of $0.3$s.
  arXiv  Detail & Related papers  (2022-06-06T11:44:43Z)
• Any-resolution Training for High-resolution Image Synthesis [55.19874755679901]
  Generative models operate at fixed resolution, even though natural images come in a variety of sizes. We argue that every pixel matters and create datasets with variable-size images, collected at their native resolutions. We introduce continuous-scale training, a process that samples patches at random scales to train a new generator with variable output resolutions.
  arXiv  Detail & Related papers  (2022-04-14T17:59:31Z)
• Improved Transformer for High-Resolution GANs [69.42469272015481]
  We introduce two key ingredients to Transformer to address this challenge. We show in the experiments that the proposed HiT achieves state-of-the-art FID scores of 31.87 and 2.95 on unconditional ImageNet $128 times 128$ and FFHQ $256 times 256$, respectively.
  arXiv  Detail & Related papers  (2021-06-14T17:39:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.