SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping

• URL: http://arxiv.org/abs/2506.08908v3
• Date: Thu, 10 Jul 2025 06:20:05 GMT
• Title: SkipVAR: Accelerating Visual Autoregressive Modeling via Adaptive Frequency-Aware Skipping
• Authors: Jiajun Li, Yue Ma, Xinyu Zhang, Qingyan Wei, Songhua Liu, Linfeng Zhang,
• Abstract summary: High-frequency components, or later steps, in the generation process contribute disproportionately to inference latency.<n>We identify two primary sources of inefficiency: step redundancy and unconditional branch redundancy.<n>We propose an automatic step-skipping strategy that selectively omits unnecessary generation steps to improve efficiency.
• Score: 30.85025293160079
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Recent studies on Visual Autoregressive (VAR) models have highlighted that high-frequency components, or later steps, in the generation process contribute disproportionately to inference latency. However, the underlying computational redundancy involved in these steps has yet to be thoroughly investigated. In this paper, we conduct an in-depth analysis of the VAR inference process and identify two primary sources of inefficiency: step redundancy and unconditional branch redundancy. To address step redundancy, we propose an automatic step-skipping strategy that selectively omits unnecessary generation steps to improve efficiency. For unconditional branch redundancy, we observe that the information gap between the conditional and unconditional branches is minimal. Leveraging this insight, we introduce unconditional branch replacement, a technique that bypasses the unconditional branch to reduce computational cost. Notably, we observe that the effectiveness of acceleration strategies varies significantly across different samples. Motivated by this, we propose SkipVAR, a sample-adaptive framework that leverages frequency information to dynamically select the most suitable acceleration strategy for each instance. To evaluate the role of high-frequency information, we introduce high-variation benchmark datasets that test model sensitivity to fine details. Extensive experiments show SkipVAR achieves over 0.88 average SSIM with up to 1.81x overall acceleration and 2.62x speedup on the GenEval benchmark, maintaining model quality. These results confirm the effectiveness of frequency-aware, training-free adaptive acceleration for scalable autoregressive image generation. Our code is available at https://github.com/fakerone-li/SkipVAR and has been publicly released.

Related papers

• SimpleGVR: A Simple Baseline for Latent-Cascaded Video Super-Resolution [55.14432034345353]
  We study key design principles for latter cascaded video super-resolution models, which are underexplored currently.<n>First, we propose two strategies to generate training pairs that better mimic the output characteristics of the base model, ensuring alignment between the VSR model and its upstream generator.<n>Second, we provide critical insights into VSR model behavior through systematic analysis of (1) timestep sampling strategies, (2) noise augmentation effects on low-resolution (LR) inputs.
  arXiv  Detail & Related papers  (2025-06-24T17:57:26Z)
• Flow-GRPO: Training Flow Matching Models via Online RL [75.70017261794422]
  We propose Flow-GRPO, the first method integrating online reinforcement learning (RL) into flow matching models.<n>Our approach uses two key strategies: (1) an ODE-to-SDE conversion that transforms a deterministic Ordinary Equation (ODE) into an equivalent Differential Equation (SDE) that matches the original model's marginal distribution at all timesteps; and (2) a Denoising Reduction strategy that reduces training denoising steps while retaining the original inference timestep number.
  arXiv  Detail & Related papers  (2025-05-08T17:58:45Z)
• Optimizing Asynchronous Federated Learning: A~Delicate Trade-Off Between Model-Parameter Staleness and Update Frequency [0.9999629695552195]
  We use modeling and analysis to better understand the impact of design choices in asynchronous FL algorithms.<n>We characterize a fundamental trade-off for optimizing asynchronous FL.<n>We show that these optimizations enhance accuracy by 10% to 30%.
  arXiv  Detail & Related papers  (2025-02-12T08:38:13Z)
• FreqMixFormerV2: Lightweight Frequency-aware Mixed Transformer for Human Skeleton Action Recognition [9.963966059349731]
  FreqMixForemrV2 is built upon the Frequency-aware Mixed Transformer (FreqMixFormer) for identifying subtle and discriminative actions.<n>The proposed model achieves a superior balance between efficiency and accuracy, outperforming state-of-the-art methods with only 60% of the parameters.
  arXiv  Detail & Related papers  (2024-12-29T23:52:40Z)
• FastSTI: A Fast Conditional Pseudo Numerical Diffusion Model for Spatio-temporal Traffic Data Imputation [4.932317347331121]
  High-temporal traffic data is crucial for intelligent transportation systems (ITS) and their data-driven applications. Recent studies of diffusion probability models have demonstrated the superiority of deep generative models in imputation. Fast on two types of real-world traffic datasets proves its ability to impute higher-quality samples in only six steps.
  arXiv  Detail & Related papers  (2024-10-20T01:45:51Z)
• REP: Resource-Efficient Prompting for Rehearsal-Free Continual Learning [23.92661395403251]
  Recent rehearsal-free methods, guided by prompts, excel in vision-related continual learning (CL) with drifting data but lack resource efficiency.<n>We introduce Resource-Efficient Prompting (REP), which improves the computational and memory efficiency of prompt-based rehearsal-free methods.<n>Our approach employs swift prompt selection to refine input data using a carefully provisioned model.
  arXiv  Detail & Related papers  (2024-06-07T09:17:33Z)
• A-SDM: Accelerating Stable Diffusion through Redundancy Removal and Performance Optimization [54.113083217869516]
  In this work, we first explore the computational redundancy part of the network. We then prune the redundancy blocks of the model and maintain the network performance. Thirdly, we propose a global-regional interactive (GRI) attention to speed up the computationally intensive attention part.
  arXiv  Detail & Related papers  (2023-12-24T15:37:47Z)
• AutoLoRa: A Parameter-Free Automated Robust Fine-Tuning Framework [13.471022394534465]
  Robust Fine-Tuning (RFT) is a low-cost strategy to obtain adversarial robustness in downstream applications. This paper uncovers an issue with the existing RFT, where optimizing both adversarial and natural objectives through the feature extractor (FE) yields significantly divergent gradient directions. We propose a low-rank (LoRa) branch that disentangles RFT into two distinct components: optimizing natural objectives via the LoRa branch and adversarial objectives via the FE.
  arXiv  Detail & Related papers  (2023-10-03T06:16:03Z)
• Fine-tuning Strategies for Faster Inference using Speech Self-Supervised Models: A Comparative Study [25.58608455210458]
  Self-supervised learning (SSL) has allowed substantial progress in Automatic Speech Recognition (ASR) performance in low-resource settings. This article explores different approaches that may be deployed during the fine-tuning to reduce the computations needed in the SSL encoder.
  arXiv  Detail & Related papers  (2023-03-12T19:52:34Z)
• Efficient Few-Shot Object Detection via Knowledge Inheritance [62.36414544915032]
  Few-shot object detection (FSOD) aims at learning a generic detector that can adapt to unseen tasks with scarce training samples. We present an efficient pretrain-transfer framework (PTF) baseline with no computational increment. We also propose an adaptive length re-scaling (ALR) strategy to alleviate the vector length inconsistency between the predicted novel weights and the pretrained base weights.
  arXiv  Detail & Related papers  (2022-03-23T06:24:31Z)
• Layer Pruning on Demand with Intermediate CTC [50.509073206630994]
  We present a training and pruning method for ASR based on the connectionist temporal classification (CTC) We show that a Transformer-CTC model can be pruned in various depth on demand, improving real-time factor from 0.005 to 0.002 on GPU.
  arXiv  Detail & Related papers  (2021-06-17T02:40:18Z)
• SUOD: Accelerating Large-Scale Unsupervised Heterogeneous Outlier Detection [63.253850875265115]
  Outlier detection (OD) is a key machine learning (ML) task for identifying abnormal objects from general samples. We propose a modular acceleration system, called SUOD, to address it.
  arXiv  Detail & Related papers  (2020-03-11T00:22:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.