FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction

• URL: http://arxiv.org/abs/2502.20313v1
• Date: Thu, 27 Feb 2025 17:39:17 GMT
• Title: FlexVAR: Flexible Visual Autoregressive Modeling without Residual Prediction
• Authors: Siyu Jiao, Gengwei Zhang, Yinlong Qian, Jiancheng Huang, Yao Zhao, Humphrey Shi, Lin Ma, Yunchao Wei, Zequn Jie,
• Abstract summary: This work challenges the residual prediction paradigm in visual autoregressive modeling.<n>It presents a new Flexible Visual AutoRegressive image generation paradigm.<n>This simple, intuitive approach swiftly learns visual distributions and makes the generation process more flexible and adaptable.
• Score: 91.09318592542509
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: This work challenges the residual prediction paradigm in visual autoregressive modeling and presents FlexVAR, a new Flexible Visual AutoRegressive image generation paradigm. FlexVAR facilitates autoregressive learning with ground-truth prediction, enabling each step to independently produce plausible images. This simple, intuitive approach swiftly learns visual distributions and makes the generation process more flexible and adaptable. Trained solely on low-resolution images ($\leq$ 256px), FlexVAR can: (1) Generate images of various resolutions and aspect ratios, even exceeding the resolution of the training images. (2) Support various image-to-image tasks, including image refinement, in/out-painting, and image expansion. (3) Adapt to various autoregressive steps, allowing for faster inference with fewer steps or enhancing image quality with more steps. Our 1.0B model outperforms its VAR counterpart on the ImageNet 256$\times$256 benchmark. Moreover, when zero-shot transfer the image generation process with 13 steps, the performance further improves to 2.08 FID, outperforming state-of-the-art autoregressive models AiM/VAR by 0.25/0.28 FID and popular diffusion models LDM/DiT by 1.52/0.19 FID, respectively. When transferring our 1.0B model to the ImageNet 512$\times$512 benchmark in a zero-shot manner, FlexVAR achieves competitive results compared to the VAR 2.3B model, which is a fully supervised model trained at 512$\times$512 resolution.

Related papers

• SimpleAR: Pushing the Frontier of Autoregressive Visual Generation through Pretraining, SFT, and RL [112.92522479863054]
  This work presents SimpleAR, a vanilla autoregressive visual generation framework without complex architecure modifications. We demonstrate that our model can generate 1024x1024 resolution images with high fidelity, and achieve competitive results on text-to-image benchmarks. By sharing these findings and open-sourcing the code, we hope to reveal the potential of autoregressive visual generation.
  arXiv  Detail & Related papers  (2025-04-15T17:59:46Z)
• SoftVQ-VAE: Efficient 1-Dimensional Continuous Tokenizer [45.720721058671856]
  SoftVQ-VAE is a continuous image tokenizer that leverages soft categorical posteriors to aggregate multiple codewords into each latent token.<n>Our approach compresses 256x256 and 512x512 images using as few as 32 or 64 1-dimensional tokens.<n>Remarkably, SoftVQ-VAE improves inference throughput by up to 18x for generating 256x256 images and 55x for 512x512 images.
  arXiv  Detail & Related papers  (2024-12-14T20:29:29Z)
• M-VAR: Decoupled Scale-wise Autoregressive Modeling for High-Quality Image Generation [39.97174784206976]
  We show that this scale-wise autoregressive framework can be effectively decoupled into textitintra-scale modeling We apply linear-complexity mechanisms like Mamba to substantially reduce computational overhead. Experiments demonstrate that our method outperforms existing models in both image quality and generation speed.
  arXiv  Detail & Related papers  (2024-11-15T18:54:42Z)
• Randomized Autoregressive Visual Generation [26.195148077398223]
  This paper presents Randomized AutoRegressive modeling (RAR) for visual generation. RAR sets a new state-of-the-art performance on the image generation task while maintaining full compatibility with language modeling frameworks. On the ImageNet-256 benchmark, RAR achieves an FID score of 1.48, not only surpassing prior state-the-art autoregressive image generators but also outperforming leading diffusion-based and masked transformer-based methods.
  arXiv  Detail & Related papers  (2024-11-01T17:59:58Z)
• Meissonic: Revitalizing Masked Generative Transformers for Efficient High-Resolution Text-to-Image Synthesis [62.06970466554273]
  We present Meissonic, which elevates non-autoregressive masked image modeling (MIM) text-to-image to a level comparable with state-of-the-art diffusion models like SDXL.<n>We leverage high-quality training data, integrate micro-conditions informed by human preference scores, and employ feature compression layers.<n>Our model not only matches but often exceeds the performance of existing models like SDXL in generating high-quality, high-resolution images.
  arXiv  Detail & Related papers  (2024-10-10T17:59:17Z)
• Autoregressive Model Beats Diffusion: Llama for Scalable Image Generation [52.509092010267665]
  We introduce LlamaGen, a new family of image generation models that apply original next-token prediction'' paradigm of large language models to visual generation domain. It is an affirmative answer to whether vanilla autoregressive models, e.g., Llama, without inductive biases on visual signals can achieve state-of-the-art image generation performance if scaling properly.
  arXiv  Detail & Related papers  (2024-06-10T17:59:52Z)
• Dynamic Pre-training: Towards Efficient and Scalable All-in-One Image Restoration [100.54419875604721]
  All-in-one image restoration tackles different types of degradations with a unified model instead of having task-specific, non-generic models for each degradation. We propose DyNet, a dynamic family of networks designed in an encoder-decoder style for all-in-one image restoration tasks. Our DyNet can seamlessly switch between its bulkier and lightweight variants, thereby offering flexibility for efficient model deployment.
  arXiv  Detail & Related papers  (2024-04-02T17:58:49Z)
• SDXS: Real-Time One-Step Latent Diffusion Models with Image Conditions [5.100085108873068]
  We present two models, SDXS-512 and SDXS-1024, achieving inference speeds of approximately 100 FPS (30x faster than SD v1.5) and 30 FPS (60x faster than SDXL) on a single GPU. Our training approach offers promising applications in image-conditioned control, facilitating efficient image-to-image translation.
  arXiv  Detail & Related papers  (2024-03-25T11:16:23Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• Emage: Non-Autoregressive Text-to-Image Generation [63.347052548210236]
  Non-autoregressive text-to-image models efficiently generate hundreds of image tokens in parallel. Our model with 346M parameters generates an image of 256$times$256 with about one second on one V100 GPU.
  arXiv  Detail & Related papers  (2023-12-22T10:01:54Z)
• Vector-quantized Image Modeling with Improved VQGAN [93.8443646643864]
  We propose a Vector-quantized Image Modeling approach that involves pretraining a Transformer to predict image tokens autoregressively. We first propose multiple improvements over vanilla VQGAN from architecture to codebook learning, yielding better efficiency and reconstruction fidelity. When trained on ImageNet at 256x256 resolution, we achieve Inception Score (IS) of 175.1 and Frechet Inception Distance (FID) of 4.17, a dramatic improvement over the vanilla VQGAN.
  arXiv  Detail & Related papers  (2021-10-09T18:36:00Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.