MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention

• URL: http://arxiv.org/abs/2506.12035v1
• Date: Thu, 22 May 2025 23:26:56 GMT
• Title: MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention
• Authors: Chaoyi Jiang, Sungwoo Kim, Lei Gao, Hossein Entezari Zarch, Won Woo Ro, Murali Annavaram,
• Abstract summary: Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation.<n>While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step.<n>We propose a training-free generation framework MARch'e to address this inefficiency through two key components: cache-aware attention and selective KV refresh.
• Score: 10.077033449956806
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation. While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step, despite most tokens remaining semantically stable across steps. We propose a training-free generation framework MARch\'e to address this inefficiency through two key components: cache-aware attention and selective KV refresh. Cache-aware attention partitions tokens into active and cached sets, enabling separate computation paths that allow efficient reuse of previously computed key/value projections without compromising full-context modeling. But a cached token cannot be used indefinitely without recomputation due to the changing contextual information over multiple steps. MARch\'e recognizes this challenge and applies a technique called selective KV refresh. Selective KV refresh identifies contextually relevant tokens based on attention scores from newly generated tokens and updates only those tokens that require recomputation, while preserving image generation quality. MARch\'e significantly reduces redundant computation in MAR without modifying the underlying architecture. Empirically, MARch\'e achieves up to 1.7x speedup with negligible impact on image quality, offering a scalable and broadly applicable solution for efficient masked transformer generation.

Related papers

• Local Representative Token Guided Merging for Text-to-Image Generation [26.585985828583304]
  Local representative token guided merging (ReToM) is a novel token merging strategy applicable to any attention mechanism in image generation.<n> Experimental results show that ReToM achieves a 6.2% improvement in FID and higher CLIP scores compared to the baseline.
  arXiv  Detail & Related papers  (2025-07-17T04:16:24Z)
• HMAR: Efficient Hierarchical Masked Auto-Regressive Image Generation [91.08481618973111]
  Visual Auto-Regressive modeling ( VAR) has shown promise in bridging the speed and quality gap between autoregressive image models and diffusion models.<n>We introduce Hierarchical Masked Auto-Regressive modeling (HMAR) to generate high-quality images with fast sampling.<n>HMAR reformulates next-scale prediction as a Markovian process, wherein the prediction of each resolution scale is conditioned only on tokens in its immediate predecessor.
  arXiv  Detail & Related papers  (2025-06-04T20:08:07Z)
• Plug-and-Play Context Feature Reuse for Efficient Masked Generation [36.563229330549284]
  Masked generative models (MGMs) have emerged as a powerful framework for image synthesis.<n>We introduce ReCAP (Reused Context-Aware Prediction), a plug-and-play module that accelerates inference in MGMs.
  arXiv  Detail & Related papers  (2025-05-25T10:57:35Z)
• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer.<n>Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis.<n>In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• Seeing What Matters: Empowering CLIP with Patch Generation-to-Selection [54.21851618853518]
  We present a concise yet effective approach called Patch Generation-to-Selection to enhance CLIP's training efficiency.<n>Our approach, CLIP-PGS, sets new state-of-the-art results in zero-shot classification and retrieval tasks.
  arXiv  Detail & Related papers  (2025-03-21T12:10:38Z)
• LazyMAR: Accelerating Masked Autoregressive Models via Feature Caching [33.024044212891326]
  Masked Autoregressive (MAR) models have emerged as a promising approach in image generation.<n>We propose LazyMAR, which introduces two caching mechanisms to handle them one by one.<n>Our method achieves 2.83 times acceleration with almost no drop in generation quality.
  arXiv  Detail & Related papers  (2025-03-16T10:54:59Z)
• Efficient Inference of Vision Instruction-Following Models with Elastic Cache [76.44955111634545]
  We introduce Elastic Cache, a novel strategy for efficient deployment of instruction-following large vision-language models. We propose an importance-driven cache merging strategy to prune redundancy caches. For instruction encoding, we utilize the frequency to evaluate the importance of caches. Results on a range of LVLMs demonstrate that Elastic Cache not only boosts efficiency but also notably outperforms existing pruning methods in language generation.
  arXiv  Detail & Related papers  (2024-07-25T15:29:05Z)
• LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation [37.72775203647514]
  This paper proposes to use learnable meta tokens to formulate sparse tokens, which effectively learn key information and improve inference speed. By employing Dual Cross-Attention (DCA) in the early stages with dense visual tokens, we obtain the hierarchical architecture LeMeViT with various sizes. Experimental results in classification and dense prediction tasks show that LeMeViT has a significant $1.7 times$ speedup, fewer parameters, and competitive performance compared to the baseline models.
  arXiv  Detail & Related papers  (2024-05-16T03:26:06Z)
• Keyformer: KV Cache Reduction through Key Tokens Selection for Efficient Generative Inference [2.8241099113277666]
  "Keyformer" is an innovative inference-time approach to mitigate the challenges associated with KV cache size and memory bandwidth utilization. We evaluate Keyformer's performance across three foundational models: GPT-J, Cerebras-GPT, and MPT.
  arXiv  Detail & Related papers  (2024-03-14T02:42:42Z)
• Rethinking Patch Dependence for Masked Autoencoders [89.02576415930963]
  We study the impact of inter-patch dependencies in the decoder of masked autoencoders (MAE) on representation learning.<n>We propose a simple visual pretraining framework: cross-attention masked autoencoders (CrossMAE)
  arXiv  Detail & Related papers  (2024-01-25T18:49:57Z)
• Masked Autoencoders Are Scalable Vision Learners [60.97703494764904]
  Masked autoencoders (MAE) are scalable self-supervised learners for computer vision. Our MAE approach is simple: we mask random patches of the input image and reconstruct the missing pixels. Coupling these two designs enables us to train large models efficiently and effectively.
  arXiv  Detail & Related papers  (2021-11-11T18:46:40Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.