SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation

• URL: http://arxiv.org/abs/2510.18716v1
• Date: Tue, 21 Oct 2025 15:17:37 GMT
• Title: SSD: Spatial-Semantic Head Decoupling for Efficient Autoregressive Image Generation
• Abstract summary: We introduce a novel KV cache compression framework for autoregressive image generation models.<n>Our experiments demonstrate that the proposed method achieves a 5$times$ reduction in memory usage and a notable 6.6$times$ speedup in overall throughput.
• Score: 5.864475030577771
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Autoregressive image generation models like Janus-Pro produce high-quality images, but at the significant cost of high memory and ever-growing computational demands due to the large number of visual tokens. While KV cache compression has been extensively studied in language modeling, it still remains largely unexplored for the image generation domain. In this work, we begin by identifying a distinct and prominent attention phenomenon, which we term spatial locality and emergent semantic sink. To leverage this key insight, we introduce a novel KV cache compression framework. Specifically, we compress the KV cache for all visual tokens by adaptively decoupling attention heads into two separate types: for spatial-locality heads, our method maintains a short recent token window; for semantic-sink heads, it strategically preserves a compact set of highly-attended tokens. Our extensive experiments demonstrate that the proposed method achieves a 5$\times$ reduction in memory usage and a notable 6.6$\times$ speedup in overall throughput with only minimal visual quality loss, thereby enabling highly efficient native autoregressive image generation on resource-constrained hardware.

Related papers

• Head-Aware Key-Value Compression for Efficient Autoregressive Image Generation [27.042998548651358]
  Autoregressive (AR) visual generation has achieved remarkable performance but suffers from high memory usage and low throughput.<n>Recent research has shown that retaining only a few lines of cache tokens can maintain high-quality images while significantly reducing memory usage and improving throughput.<n>We propose a novel head-aware key-value (KV) cache compression framework for autoregressive image generation, called HeadKV.
  arXiv  Detail & Related papers  (2026-05-20T01:30:33Z)
• Reducing Peak Memory Usage for Modern Multimodal Large Language Model Pipelines [4.564896252406974]
  Multimodal large language models (MLLMs) have recently demonstrated strong capabilities in understanding and generating responses from diverse visual inputs.<n>As these models scale to richer visual representations, inference increasingly relies on storing large numbers of vision tokens in the key-value cache.<n>We show that MLLMs exhibit inherent structural regularities and representational redundancy that can be exploited to control memory growth throughout inference.
  arXiv  Detail & Related papers  (2026-04-17T22:45:57Z)
• HybridKV: Hybrid KV Cache Compression for Efficient Multimodal Large Language Model Inference [23.016627145448467]
  Multimodal Large Language Models (MLLMs) have advanced unified reasoning over text, images, and videos.<n>Each visual input expands into thousands of tokens, causing caches to scale linearly with context length.<n>A common solution is to compress caches under a fixed allocated budget at different granularities.<n>We propose HybridKV, a hybrid KV cache compression framework that integrates complementary strategies in three stages.
  arXiv  Detail & Related papers  (2026-04-07T13:51:07Z)
• Autoregressive Image Generation Needs Only a Few Lines of Cached Tokens [33.3294598877681]
  LineAR is a novel, training-free progressive key-value ( KV) cache compression pipeline for autoregressive image generation.<n>LineAR manages the cache at the line level using a 2D view, preserving the visual dependency regions while evicting less-informative tokens.<n>LineAR achieves significant memory and throughput gains, including up to 67.61% memory reduction and 7.57x speedup.
  arXiv  Detail & Related papers  (2025-12-04T14:41:21Z)
• GUI-KV: Efficient GUI Agents via KV Cache with Spatio-Temporal Awareness [75.00019285120878]
  Key-value (KV) caching can mitigate this, but storing the full cache is prohibitive for image-heavy contexts.<n>Existing cache-compression methods are sub-optimal as they do not account for the spatial and temporal redundancy of GUIs.<n>We introduce GUI-KV, a plug-and-play KV cache compression method for GUI agents that requires no retraining.
  arXiv  Detail & Related papers  (2025-10-01T05:37:54Z)
• MARché: Fast Masked Autoregressive Image Generation with Cache-Aware Attention [10.077033449956806]
  Masked autoregressive (MAR) models unify the strengths of masked and autoregressive generation by predicting tokens in a fixed order using bidirectional attention for image generation.<n>While effective, MAR models suffer from significant computational overhead, as they recompute attention and feed-forward representations for all tokens at every decoding step.<n>We propose a training-free generation framework MARch'e to address this inefficiency through two key components: cache-aware attention and selective KV refresh.
  arXiv  Detail & Related papers  (2025-05-22T23:26:56Z)
• Token-Shuffle: Towards High-Resolution Image Generation with Autoregressive Models [92.18057318458528]
  Token-Shuffle is a novel method that reduces the number of image tokens in Transformer.<n>Our strategy requires no additional pretrained text-encoder and enables MLLMs to support extremely high-resolution image synthesis.<n>In GenAI-benchmark, our 2.7B model achieves 0.77 overall score on hard prompts, outperforming AR models LlamaGen by 0.18 and diffusion models LDM by 0.15.
  arXiv  Detail & Related papers  (2025-04-24T17:59:56Z)
• Efficient LLaMA-3.2-Vision by Trimming Cross-attended Visual Features [24.33252753245426]
  We exploit the sparse nature in cross-attention maps to selectively prune redundant visual features.<n>Our model can reduce inference latency and memory usage while achieving benchmark parity.
  arXiv  Detail & Related papers  (2025-04-01T09:10:32Z)
• CSR:Achieving 1 Bit Key-Value Cache via Sparse Representation [63.65323577445951]
  We propose a novel approach called Cache Sparse Representation (CSR)<n>CSR transforms the dense Key-Value cache tensor into sparse indexes and weights, offering a more memory-efficient representation during LLM inference.<n>Our experiments demonstrate CSR achieves performance comparable to state-of-the-art KV cache quantization algorithms.
  arXiv  Detail & Related papers  (2024-12-16T13:01:53Z)
• Collaborative Decoding Makes Visual Auto-Regressive Modeling Efficient [52.96232442322824]
  Collaborative Decoding (CoDe) is a novel efficient decoding strategy tailored for the Visual Auto-Regressive ( VAR) framework.<n>CoDe capitalizes on two critical observations: the substantially reduced parameter demands at larger scales and the exclusive generation patterns across different scales.<n>CoDe achieves a 1.7x speedup, slashes memory usage by around 50%, and preserves image quality with only a negligible FID increase from 1.95 to 1.98.
  arXiv  Detail & Related papers  (2024-11-26T15:13:15Z)
• ImageFolder: Autoregressive Image Generation with Folded Tokens [51.815319504939396]
  Increasing token length is a common approach to improve the image reconstruction quality.<n>There exists a trade-off between reconstruction and generation quality regarding token length.<n>We propose Image, a semantic tokenizer that provides spatially aligned image tokens that can be folded during autoregressive modeling.
  arXiv  Detail & Related papers  (2024-10-02T17:06:39Z)
• HINT: High-quality INPainting Transformer with Mask-Aware Encoding and Enhanced Attention [14.055584700641212]
  Existing image inpainting methods leverage convolution-based downsampling approaches to reduce spatial dimensions. We propose an end-to-end High-quality INpainting Transformer, abbreviated as HINT, which consists of a novel mask-aware pixel-shuffle downsampling module. We demonstrate the superior performance of HINT compared to contemporary state-of-the-art models on four datasets.
  arXiv  Detail & Related papers  (2024-02-22T00:14:26Z)
• LoMA: Lossless Compressed Memory Attention [0.0]
  Lossless Compressed Memory Attention (LoMA) is a novel approach to reduce memory and computational demands during autoregressive generation. LoMA incorporates a specialized training or fine-tuning precedure alongside an autoregressive generation algorithm optimized for the compressed context. Experimental validation has demonstrated that LoMA significantly reducing computational consumption and memory usage.
  arXiv  Detail & Related papers  (2024-01-16T09:18:46Z)
• Model Tells You What to Discard: Adaptive KV Cache Compression for LLMs [82.08922896531618]
  We introduce adaptive KV cache compression, a plug-and-play method that reduces the memory footprint of generative inference for Large Language Models (LLMs) We conduct targeted profiling to discern the intrinsic structure of attention modules. Based on the recognized structure, we then construct the KV cache in an adaptive manner: evicting long-range contexts on attention heads emphasizing local contexts, discarding non-special tokens on attention heads centered on special tokens, and only employing the standard KV cache for attention heads that broadly attend to all tokens.
  arXiv  Detail & Related papers  (2023-10-03T05:17:08Z)
• Fast and High-Quality Image Denoising via Malleable Convolutions [72.18723834537494]
  We present Malleable Convolution (MalleConv), as an efficient variant of dynamic convolution. Unlike previous works, MalleConv generates a much smaller set of spatially-varying kernels from input. We also build an efficient denoising network using MalleConv, coined as MalleNet.
  arXiv  Detail & Related papers  (2022-01-02T18:35:20Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.