Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis

• URL: http://arxiv.org/abs/2406.05478v1
• Date: Sat, 8 Jun 2024 13:52:20 GMT
• Title: Revisiting Non-Autoregressive Transformers for Efficient Image Synthesis
• Authors: Zanlin Ni, Yulin Wang, Renping Zhou, Jiayi Guo, Jinyi Hu, Zhiyuan Liu, Shiji Song, Yuan Yao, Gao Huang,
• Abstract summary: Non-autoregressive Transformers (NATs) have been recognized for their rapid generation. We re-evaluate the full potential of NATs by revisiting the design of their training and inference strategies. We propose to go beyond existing methods by directly solving the optimal strategies in an automatic framework.
• Score: 82.72941975704374
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The field of image synthesis is currently flourishing due to the advancements in diffusion models. While diffusion models have been successful, their computational intensity has prompted the pursuit of more efficient alternatives. As a representative work, non-autoregressive Transformers (NATs) have been recognized for their rapid generation. However, a major drawback of these models is their inferior performance compared to diffusion models. In this paper, we aim to re-evaluate the full potential of NATs by revisiting the design of their training and inference strategies. Specifically, we identify the complexities in properly configuring these strategies and indicate the possible sub-optimality in existing heuristic-driven designs. Recognizing this, we propose to go beyond existing methods by directly solving the optimal strategies in an automatic framework. The resulting method, named AutoNAT, advances the performance boundaries of NATs notably, and is able to perform comparably with the latest diffusion models at a significantly reduced inference cost. The effectiveness of AutoNAT is validated on four benchmark datasets, i.e., ImageNet-256 & 512, MS-COCO, and CC3M. Our code is available at https://github.com/LeapLabTHU/ImprovedNAT.

Related papers

• DiP-GO: A Diffusion Pruner via Few-step Gradient Optimization [22.546989373687655]
  We propose a novel pruning method that derives an efficient diffusion model via a more intelligent and differentiable pruner. Our approach achieves 4.4 x speedup for SD-1.5 without any loss of accuracy, significantly outperforming the previous state-of-the-art methods.
  arXiv  Detail & Related papers  (2024-10-22T12:18:24Z)
• Diffusion Models as Optimizers for Efficient Planning in Offline RL [47.0835433289033]
  Diffusion models have shown strong competitiveness in offline reinforcement learning tasks. We propose a faster autoregressive model to handle the generation of feasible trajectories. This allows us to achieve more efficient planning without sacrificing capability.
  arXiv  Detail & Related papers  (2024-07-23T03:00:01Z)
• Binarized Diffusion Model for Image Super-Resolution [61.963833405167875]
  Binarization, an ultra-compression algorithm, offers the potential for effectively accelerating advanced diffusion models (DMs) Existing binarization methods result in significant performance degradation. We introduce a novel binarized diffusion model, BI-DiffSR, for image SR.
  arXiv  Detail & Related papers  (2024-06-09T10:30:25Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• Optimizing Non-Autoregressive Transformers with Contrastive Learning [74.46714706658517]
  Non-autoregressive Transformers (NATs) reduce the inference latency of Autoregressive Transformers (ATs) by predicting words all at once rather than in sequential order. In this paper, we propose to ease the difficulty of modality learning via sampling from the model distribution instead of the data distribution.
  arXiv  Detail & Related papers  (2023-05-23T04:20:13Z)
• RenewNAT: Renewing Potential Translation for Non-Autoregressive Transformer [15.616188012177538]
  Non-autoregressive neural machine translation (NAT) models are proposed to accelerate the inference process while maintaining relatively high performance. Existing NAT models are difficult to achieve the desired efficiency-quality trade-off. We propose RenewNAT, a flexible framework with high efficiency and effectiveness.
  arXiv  Detail & Related papers  (2023-03-14T07:10:03Z)
• TSNAT: Two-Step Non-Autoregressvie Transformer Models for Speech Recognition [69.68154370877615]
  The non-autoregressive (NAR) models can get rid of the temporal dependency between the output tokens and predict the entire output tokens in at least one step. To address these two problems, we propose a new model named the two-step non-autoregressive transformer(TSNAT) The results show that the TSNAT can achieve a competitive performance with the AR model and outperform many complicated NAR models.
  arXiv  Detail & Related papers  (2021-04-04T02:34:55Z)
• Fully Non-autoregressive Neural Machine Translation: Tricks of the Trade [47.97977478431973]
  Fully non-autoregressive neural machine translation (NAT) is proposed to simultaneously predict tokens with single forward of neural networks. In this work, we target on closing the performance gap while maintaining the latency advantage.
  arXiv  Detail & Related papers  (2020-12-31T18:52:59Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.