DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation

• URL: http://arxiv.org/abs/2410.08159v2
• Date: Thu, 23 Jan 2025 17:08:57 GMT
• Title: DART: Denoising Autoregressive Transformer for Scalable Text-to-Image Generation
• Authors: Jiatao Gu, Yuyang Wang, Yizhe Zhang, Qihang Zhang, Dinghuai Zhang, Navdeep Jaitly, Josh Susskind, Shuangfei Zhai,
• Abstract summary: Diffusion models are trained by denoising a Markovian process which gradually adds noise to the input. We propose DART, a transformer-based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework.
• Score: 46.5013105017258
• License:
• Abstract: Diffusion models have become the dominant approach for visual generation. They are trained by denoising a Markovian process which gradually adds noise to the input. We argue that the Markovian property limits the model's ability to fully utilize the generation trajectory, leading to inefficiencies during training and inference. In this paper, we propose DART, a transformer-based model that unifies autoregressive (AR) and diffusion within a non-Markovian framework. DART iteratively denoises image patches spatially and spectrally using an AR model that has the same architecture as standard language models. DART does not rely on image quantization, which enables more effective image modeling while maintaining flexibility. Furthermore, DART seamlessly trains with both text and image data in a unified model. Our approach demonstrates competitive performance on class-conditioned and text-to-image generation tasks, offering a scalable, efficient alternative to traditional diffusion models. Through this unified framework, DART sets a new benchmark for scalable, high-quality image synthesis.

Related papers

• One Diffusion Step to Real-World Super-Resolution via Flow Trajectory Distillation [60.54811860967658]
  FluxSR is a novel one-step diffusion Real-ISR based on flow matching models. First, we introduce Flow Trajectory Distillation (FTD) to distill a multi-step flow matching model into a one-step Real-ISR. Second, to improve image realism and address high-frequency artifact issues in generated images, we propose TV-LPIPS as a perceptual loss.
  arXiv  Detail & Related papers  (2025-02-04T04:11:29Z)
• MMAR: Towards Lossless Multi-Modal Auto-Regressive Probabilistic Modeling [64.09238330331195]
  We propose a novel Multi-Modal Auto-Regressive (MMAR) probabilistic modeling framework. Unlike discretization line of method, MMAR takes in continuous-valued image tokens to avoid information loss. We show that MMAR demonstrates much more superior performance than other joint multi-modal models.
  arXiv  Detail & Related papers  (2024-10-14T17:57:18Z)
• TimeDART: A Diffusion Autoregressive Transformer for Self-Supervised Time Series Representation [47.58016750718323]
  We propose TimeDART, a novel self-supervised time series pre-training framework. TimeDART unifies two powerful generative paradigms to learn more transferable representations. We conduct extensive experiments on public datasets for time series forecasting and classification.
  arXiv  Detail & Related papers  (2024-10-08T06:08:33Z)
• FBSDiff: Plug-and-Play Frequency Band Substitution of Diffusion Features for Highly Controllable Text-Driven Image Translation [19.65838242227773]
  This paper contributes a novel, concise, and efficient approach that adapts pre-trained large-scale text-to-image (T2I) diffusion model to the image-to-image (I2I) paradigm in a plug-and-play manner. Our method allows flexible control over both guiding factor and guiding intensity of the reference image simply by tuning the type and bandwidth of the substituted frequency band.
  arXiv  Detail & Related papers  (2024-08-02T04:13:38Z)
• Denoising Autoregressive Representation Learning [13.185567468951628]
  Our method, DARL, employs a decoder-only Transformer to predict image patches autoregressively. We show that the learned representation can be improved by using tailored noise schedules and longer training in larger models.
  arXiv  Detail & Related papers  (2024-03-08T10:19:00Z)
• Unlocking Pre-trained Image Backbones for Semantic Image Synthesis [29.688029979801577]
  We propose a new class of GAN discriminators for semantic image synthesis that generates highly realistic images. Our model, which we dub DP-SIMS, achieves state-of-the-art results in terms of image quality and consistency with the input label maps on ADE-20K, COCO-Stuff, and Cityscapes.
  arXiv  Detail & Related papers  (2023-12-20T09:39:19Z)
• DiffiT: Diffusion Vision Transformers for Image Generation [88.08529836125399]
  Vision Transformer (ViT) has demonstrated strong modeling capabilities and scalability, especially for recognition tasks. We study the effectiveness of ViTs in diffusion-based generative learning and propose a new model denoted as Diffusion Vision Transformers (DiffiT) DiffiT is surprisingly effective in generating high-fidelity images with significantly better parameter efficiency.
  arXiv  Detail & Related papers  (2023-12-04T18:57:01Z)
• DiffDis: Empowering Generative Diffusion Model with Cross-Modal Discrimination Capability [75.9781362556431]
  We propose DiffDis to unify the cross-modal generative and discriminative pretraining into one single framework under the diffusion process. We show that DiffDis outperforms single-task models on both the image generation and the image-text discriminative tasks.
  arXiv  Detail & Related papers  (2023-08-18T05:03:48Z)
• StraIT: Non-autoregressive Generation with Stratified Image Transformer [63.158996766036736]
  Stratified Image Transformer(StraIT) is a pure non-autoregressive(NAR) generative model. Our experiments demonstrate that StraIT significantly improves NAR generation and out-performs existing DMs and AR methods.
  arXiv  Detail & Related papers  (2023-03-01T18:59:33Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.