RandAR: Decoder-only Autoregressive Visual Generation in Random Orders

• URL: http://arxiv.org/abs/2412.01827v1
• Date: Mon, 02 Dec 2024 18:59:53 GMT
• Title: RandAR: Decoder-only Autoregressive Visual Generation in Random Orders
• Authors: Ziqi Pang, Tianyuan Zhang, Fujun Luan, Yunze Man, Hao Tan, Kai Zhang, William T. Freeman, Yu-Xiong Wang,
• Abstract summary: RandAR is a decoder-only visual autoregressive (AR) model capable of generating images in arbitrary token orders.<n>Our design enables random order by inserting a "position instruction token" before each image token to be predicted.<n>RandAR supports inpainting, outpainting and resolution inference in a zero-shot manner.
• Score: 54.49937384788739
• License: http://creativecommons.org/licenses/by-nc-sa/4.0/
• Abstract: We introduce RandAR, a decoder-only visual autoregressive (AR) model capable of generating images in arbitrary token orders. Unlike previous decoder-only AR models that rely on a predefined generation order, RandAR removes this inductive bias, unlocking new capabilities in decoder-only generation. Our essential design enables random order by inserting a "position instruction token" before each image token to be predicted, representing the spatial location of the next image token. Trained on randomly permuted token sequences -- a more challenging task than fixed-order generation, RandAR achieves comparable performance to its conventional raster-order counterpart. More importantly, decoder-only transformers trained from random orders acquire new capabilities. For the efficiency bottleneck of AR models, RandAR adopts parallel decoding with KV-Cache at inference time, enjoying 2.5x acceleration without sacrificing generation quality. Additionally, RandAR supports inpainting, outpainting and resolution extrapolation in a zero-shot manner. We hope RandAR inspires new directions for decoder-only visual generation models and broadens their applications across diverse scenarios. Our project page is at https://rand-ar.github.io/.

Related papers

• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer. Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis. In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning [66.5214586624095]
  Existing Visual Autoregressive ( VAR) paradigms process the entire token map at each scale step, leading to the complexity and runtime scaling dramatically with image resolution. We propose Fastmore, a post-training acceleration method for efficient resolution scaling with VARs. Experiments show Fastmore can further speedup FlashAttention-accelerated VAR by 2.7$times$ with negligible performance drop of 1%.
  arXiv  Detail & Related papers  (2025-03-30T08:51:19Z)
• Autoregressive Image Generation with Randomized Parallel Decoding [23.714192351237628]
  ARPG is a novel visual autoregressive model that enables randomized parallel generation. Our approach attains an FID of 1.94 with only 64 sampling steps, achieving over a 20-fold increase in throughput.
  arXiv  Detail & Related papers  (2025-03-13T17:19:51Z)
• Beyond Next-Token: Next-X Prediction for Autoregressive Visual Generation [34.112157859384645]
  Autoregressive (AR) modeling underpins state-of-the-art language and visual generative models. Traditionally, a token'' is treated as the smallest prediction unit, often a discrete symbol in language or a quantized patch in vision. We propose xAR, a framework that extends the notion of a token to an entity X.
  arXiv  Detail & Related papers  (2025-02-27T18:59:08Z)
• DiCoDe: Diffusion-Compressed Deep Tokens for Autoregressive Video Generation with Language Models [72.24305287508474]
  We introduce DiCoDe, a novel approach to generate videos with a language model in an autoregressive manner. By treating videos as temporal sequences, DiCoDe fully harnesses the capabilities of language models for autoregressive generation. We evaluate DiCoDe both quantitatively and qualitatively, demonstrating that it performs comparably to existing methods in terms of quality.
  arXiv  Detail & Related papers  (2024-12-05T18:57:06Z)
• ZipAR: Accelerating Auto-regressive Image Generation through Spatial Locality [19.486745219466666]
  ZipAR is a training-free, plug-and-play parallel decoding framework for auto-regressive (AR) visual generation. ZipAR can reduce the number of model forward passes by up to 91% on the Emu3-Gen model without requiring any additional retraining.
  arXiv  Detail & Related papers  (2024-12-05T10:57:08Z)
• Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient [52.96232442322824]
  Collaborative Decoding (CoDe) is a novel efficient decoding strategy tailored for the Visual Auto-Regressive ( VAR) framework.<n>CoDe capitalizes on two critical observations: the substantially reduced parameter demands at larger scales and the exclusive generation patterns across different scales.<n>CoDe achieves a 1.7x speedup, slashes memory usage by around 50%, and preserves image quality with only a negligible FID increase from 1.95 to 1.98.
  arXiv  Detail & Related papers  (2024-11-26T15:13:15Z)
• Randomized Autoregressive Visual Generation [26.195148077398223]
  This paper presents Randomized AutoRegressive modeling (RAR) for visual generation. RAR sets a new state-of-the-art performance on the image generation task while maintaining full compatibility with language modeling frameworks. On the ImageNet-256 benchmark, RAR achieves an FID score of 1.48, not only surpassing prior state-the-art autoregressive image generators but also outperforming leading diffusion-based and masked transformer-based methods.
  arXiv  Detail & Related papers  (2024-11-01T17:59:58Z)
• LANTERN: Accelerating Visual Autoregressive Models with Relaxed Speculative Decoding [30.630803933771865]
  Experimental results demonstrate the efficacy of our method in providing a substantial speed-up over speculative decoding. LANTERN increases speed-ups by $mathbf1.75times$ and $mathbf1.76times$, as compared to greedy decoding and random sampling.
  arXiv  Detail & Related papers  (2024-10-04T12:21:03Z)
• Visual Autoregressive Modeling: Scalable Image Generation via Next-Scale Prediction [33.57820997288788]
  We present a new generation paradigm that redefines the autoregressive learning on images as coarse-to-fine "next-scale prediction" Visual AutoRegressive modeling makes GPT-like AR models surpass diffusion transformers in image generation. We have released all models and codes to promote the exploration of AR/token models for visual generation and unified learning.
  arXiv  Detail & Related papers  (2024-04-03T17:59:53Z)
• Unified Sequence-to-Sequence Learning for Single- and Multi-Modal Visual Object Tracking [64.28025685503376]
  SeqTrack casts visual tracking as a sequence generation task, forecasting object bounding boxes in an autoregressive manner. SeqTrackv2 integrates a unified interface for auxiliary modalities and a set of task-prompt tokens to specify the task. This sequence learning paradigm not only simplifies the tracking framework, but also showcases superior performance across 14 challenging benchmarks.
  arXiv  Detail & Related papers  (2023-04-27T17:56:29Z)
• Paraformer: Fast and Accurate Parallel Transformer for Non-autoregressive End-to-End Speech Recognition [62.83832841523525]
  We propose a fast and accurate parallel transformer, termed Paraformer. It accurately predicts the number of output tokens and extract hidden variables. It can attain comparable performance to the state-of-the-art AR transformer, with more than 10x speedup.
  arXiv  Detail & Related papers  (2022-06-16T17:24:14Z)
• Match What Matters: Generative Implicit Feature Replay for Continual Learning [0.0]
  We propose GenIFeR (Generative Implicit Feature Replay) for class-incremental learning. The main idea is to train a generative adversarial network (GAN) to generate images that contain realistic features. We empirically show that GenIFeR is superior to both conventional generative image and feature replay.
  arXiv  Detail & Related papers  (2021-06-09T19:29:41Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.