Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens

• URL: http://arxiv.org/abs/2512.04857v1
• Date: Thu, 04 Dec 2025 14:41:21 GMT
• Title: Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens
• Abstract summary: LineAR is a novel, training-free progressive key-value ( KV) cache compression pipeline for autoregressive image generation.<n>LineAR manages the cache at the line level using a 2D view, preserving the visual dependency regions while evicting less-informative tokens.<n>LineAR achieves significant memory and throughput gains, including up to 67.61% memory reduction and 7.57x speedup.
• Score: 33.3294598877681
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoregressive (AR) visual generation has emerged as a powerful paradigm for image and multimodal synthesis, owing to its scalability and generality. However, existing AR image generation suffers from severe memory bottlenecks due to the need to cache all previously generated visual tokens during decoding, leading to both high storage requirements and low throughput. In this paper, we introduce \textbf{LineAR}, a novel, training-free progressive key-value (KV) cache compression pipeline for autoregressive image generation. By fully exploiting the intrinsic characteristics of visual attention, LineAR manages the cache at the line level using a 2D view, preserving the visual dependency regions while progressively evicting less-informative tokens that are harmless for subsequent line generation, guided by inter-line attention. LineAR enables efficient autoregressive (AR) image generation by utilizing only a few lines of cache, achieving both memory savings and throughput speedup, while maintaining or even improving generation quality. Extensive experiments across six autoregressive image generation models, including class-conditional and text-to-image generation, validate its effectiveness and generality. LineAR improves ImageNet FID from 2.77 to 2.68 and COCO FID from 23.85 to 22.86 on LlamaGen-XL and Janus-Pro-1B, while retaining only 1/6 KV cache. It also improves DPG on Lumina-mGPT-768 with just 1/8 KV cache. Additionally, LineAR achieves significant memory and throughput gains, including up to 67.61% memory reduction and 7.57x speedup on LlamaGen-XL, and 39.66% memory reduction and 5.62x speedup on Janus-Pro-7B.

Related papers

• Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation [27.042998548651358]
  Autoregressive (AR) visual generation has achieved remarkable performance but suffers from high memory usage and low throughput.<n>Recent research has shown that retaining only a few lines of cache tokens can maintain high-quality images while significantly reducing memory usage and improving throughput.<n>We propose a novel head-aware key-value (KV) cache compression framework for autoregressive image generation, called HeadKV.
  arXiv  Detail & Related papers  (2026-05-20T01:30:33Z)
• Quant VideoGen: Auto-Regressive Long Video Generation via 2-Bit KV-Cache Quantization [83.406036390582]
  Quant VideoGen (QVG) is a training free KV cache quantization framework for autoregressive video diffusion models.<n>It reduces KV memory by up to 7.0 times with less than 4% end to end latency overhead.<n>It consistently outperforms existing baselines in generation quality.
  arXiv  Detail & Related papers  (2026-02-03T00:54:32Z)
• PackCache: A Training-Free Acceleration Method for Unified Autoregressive Video Generation via Compact KV-Cache [61.57938553036056]
  We introduce PackCache, a training-free KV-cache management method that compacts the KV cache through three coordinated mechanisms.<n>In terms of efficiency, PackCache accelerates end-to-end generation by 1.7-2.2x on 48-frame long sequences.
  arXiv  Detail & Related papers  (2026-01-07T19:51:06Z)
• DPAR: Dynamic Patchification for Efficient Autoregressive Visual Generation [10.719563134726057]
  We present DPAR, a decoder-only autoregressive model that aggregates image tokens into a variable number of patches for efficient image generation.<n>DPAR reduces token count by 1.81x and 2.06x on Imagenet 256 and 384 generation resolution respectively, leading to a reduction of up to 40% FLOPs in training costs.
  arXiv  Detail & Related papers  (2025-12-26T05:03:47Z)
• VA-$π$: Variational Policy Alignment for Pixel-Aware Autoregressive Generation [65.22452273252428]
  VA-$$ is a post-training framework to optimize autoregressive visual generation.<n>It unifies pixel reconstruction and autoregressive modeling.<n>It reduces FID from 14.36 to 7.65 and improves IS from 86.55 to 116.70 on LlamaGen-XXL.
  arXiv  Detail & Related papers  (2025-12-22T18:54:30Z)
• SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation [5.864475030577771]
  We introduce a novel KV cache compression framework for autoregressive image generation models.<n>Our experiments demonstrate that the proposed method achieves a 5$times$ reduction in memory usage and a notable 6.6$times$ speedup in overall throughput.
  arXiv  Detail & Related papers  (2025-10-21T15:17:37Z)
• MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention [10.077033449956806]
  Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation.<n>While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step.<n>We propose a training-free generation framework MARch'e to address this inefficiency through two key components: cache-aware attention and selective KV refresh.
  arXiv  Detail & Related papers  (2025-05-22T23:26:56Z)
• FastVAR: Linear Visual Autoregressive Modeling via Cached Token Pruning [66.5214586624095]
  Existing Visual Autoregressive ( VAR) paradigms process the entire token map at each scale step, leading to the complexity and runtime scaling dramatically with image resolution.<n>We propose Fastmore, a post-training acceleration method for efficient resolution scaling with VARs.<n> Experiments show Fastmore can further speedup FlashAttention-accelerated VAR by 2.7$times$ with negligible performance drop of 1%.
  arXiv  Detail & Related papers  (2025-03-30T08:51:19Z)
• PrefixKV: Adaptive Prefix KV Cache is What Vision Instruction-Following Models Need for Efficient Generation [97.41972925670508]
  Large vision-language models (LVLMs) incur significant computational and memory overhead during inference.<n>We present PrefixKV, where "Prefix" means the top-ranked KV based on importance rather than position in the original sequence.<n>Our method achieves the state-of-the-art performance compared with others.
  arXiv  Detail & Related papers  (2024-12-04T15:48:59Z)
• Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient [52.96232442322824]
  Collaborative Decoding (CoDe) is a novel efficient decoding strategy tailored for the Visual Auto-Regressive ( VAR) framework.<n>CoDe capitalizes on two critical observations: the substantially reduced parameter demands at larger scales and the exclusive generation patterns across different scales.<n>CoDe achieves a 1.7x speedup, slashes memory usage by around 50%, and preserves image quality with only a negligible FID increase from 1.95 to 1.98.
  arXiv  Detail & Related papers  (2024-11-26T15:13:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.