Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding

• URL: http://arxiv.org/abs/2608.08832v1
• Date: Sun, 09 Aug 2026 17:32:04 GMT
• Title: Visual Token Codec: Unleashing Spatial Redundancy for ViT Feature Coding
• Abstract summary: Visual Token Codec separates global and patch tokens into dedicated coding paths.<n>We show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks.
• Score: 49.540751364954666
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Distributed deployment of large vision foundation models often partitions a ViT backbone and exchanges intermediate token features between computing nodes, making efficient feature compression critical under bandwidth and computation constraints. Existing ViT feature codecs typically flatten heterogeneous global and patch tokens into an L x C pseudo image, causing entropy models to mainly capture sequence-axis dependencies while overlooking the native two-dimensional patch-grid structure. In this paper, we show that ViT patch tokens retain strong local spatial correlations on the original grid. To exploit this structural prior, we propose the Visual Token Codec (VTC), a dual-path learned codec that separates global and patch tokens into dedicated coding paths. Global tokens are compressed with a lightweight factorized prior, whereas patch tokens are encoded on the patch-token grid using a spatial-channel context entropy model. To support intermediate-layer compression and practical rate adaptation, VTC further incorporates feature-matching supervision after subsequent ViT blocks and variable-rate modules within a single codec. Experiments on DINOv2 and SAM3 show that VTC consistently outperforms representative ViT feature coding baselines on classification, segmentation, and detection tasks. At 90% of uncompressed-feature performance, VTC reduces bitrate by 15.7x-37.4x across these tasks. We further provide intermediate-layer rate-utility analyses for practical transmission- and storage-oriented deployment scenarios.

Related papers

• Compressing AI Traffic: Standardized Neural Network Coding of Visual-Token Representations in Split Vision-Language Inference [15.357950338341285]
  Video-MME accuracy remains close to the uncompressed reference up to a 98% reduction of the transmitted BF16 tensor and only then collapses.<n>Open-ended MLVU generation shows the same plateau-and- robustness profile under an LLM judge.
  arXiv  Detail & Related papers  (2026-09-01T13:07:32Z)
• PACE: A Unified Condense-and-Extract Paradigm for Fast VLM Inference [15.233755808671871]
  Existing visual token pruning methods exhibit two fundamental limitations.<n>Most approaches operate exclusively post-vision encoder, leaving the substantial latency of the visual encoding phase unoptimized.<n>We propose PACE, a framework that accelerates both the vision encoder and the Large Language Model.
  arXiv  Detail & Related papers  (2026-08-27T14:52:09Z)
• Aggregating Visual Information with Optimal Transport for VideoLM Token Compression [55.867497983465476]
  Video language models process videos as dense visual-token sequences with substantial representational redundancy.<n>These sequences are therefore essential for reducing the visual-token burden on language-model decoding.<n>We introduce Aggregating Visual Information with Optimal Transport (AVIOT), which casts video token compression as transporting a dense empirical measure of frame observations.
  arXiv  Detail & Related papers  (2026-08-20T18:05:42Z)
• Accelerating Streaming Video Large Language Models via Hierarchical Token Compression [12.247532124314402]
  Streaming Video Large Language Models (VideoLLMs) have demonstrated impressive performance across various video understanding tasks.<n>They face significant challenges in real-time deployment due to the high computational cost of processing dense visual tokens from continuous video streams.<n>We propose textbfStreaming textbfToken textbfCompression (textbfSTC), a plug-and-play hierarchical framework that seamlessly integrates into existing streaming VideoLLMs.
  arXiv  Detail & Related papers  (2025-11-30T13:44:28Z)
• LLaVA-UHD v3: Progressive Visual Compression for Efficient Native-Resolution Encoding in MLLMs [52.24096832965001]
  We present LLaVA-UHD v3, an MLLM centered upon our proposed Progressive Visual Compression (PVC) method.<n>The PVC method can be seamlessly integrated into standard Vision Transformer (ViT) to enable efficient native-resolution encoding.<n>Building upon ViT-UHD, LLaVA-UHD v3 also achieves competitive performance to Qwen2-VL, while further reducing TTFT by 1.9x.
  arXiv  Detail & Related papers  (2025-11-26T08:11:10Z)
• SCASeg: Strip Cross-Attention for Efficient Semantic Segmentation [37.2240333333522]
  Vision Transformer (ViT) has achieved notable success in computer vision, with its variants extensively validated across various downstream tasks, including semantic segmentation. This paper proposes Strip Cross-Attention (SCASeg), an innovative decoder head explicitly designed for semantic segmentation.
  arXiv  Detail & Related papers  (2024-11-26T03:00:09Z)
• High-Efficiency Neural Video Compression via Hierarchical Predictive Learning [27.41398149573729]
  Enhanced Deep Hierarchical Video Compression-DHVC 2.0- introduces superior compression performance and impressive complexity efficiency. Uses hierarchical predictive coding to transform each video frame into multiscale representations. Supports transmission-friendly progressive decoding, making it particularly advantageous for networked video applications in the presence of packet loss.
  arXiv  Detail & Related papers  (2024-10-03T15:40:58Z)
• CAIT: Triple-Win Compression towards High Accuracy, Fast Inference, and Favorable Transferability For ViTs [89.79139531731637]
  Vision Transformers (ViTs) have emerged as state-of-the-art models for various vision tasks.<n>We propose a joint underlinecompression method for ViTs that achieves a harmonious blend of high underlineaccuracy, fast underlineinference speed, and favorable underlinetransferability to downstream tasks.
  arXiv  Detail & Related papers  (2023-09-27T16:12:07Z)
• SegViTv2: Exploring Efficient and Continual Semantic Segmentation with Plain Vision Transformers [76.13755422671822]
  This paper investigates the capability of plain Vision Transformers (ViTs) for semantic segmentation using the encoder-decoder framework. We introduce a novel Attention-to-Mask (atm) module to design a lightweight decoder effective for plain ViT. Our decoder outperforms the popular decoder UPerNet using various ViT backbones while consuming only about $5%$ of the computational cost.
  arXiv  Detail & Related papers  (2023-06-09T22:29:56Z)
• SegViT: Semantic Segmentation with Plain Vision Transformers [91.50075506561598]
  We explore the capability of plain Vision Transformers (ViTs) for semantic segmentation. We propose the Attention-to-Mask (ATM) module, in which similarity maps between a set of learnable class tokens and the spatial feature maps are transferred to the segmentation masks. Experiments show that our proposed SegVit using the ATM module outperforms its counterparts using the plain ViT backbone.
  arXiv  Detail & Related papers  (2022-10-12T00:30:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.