Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice

• URL: http://arxiv.org/abs/2605.16384v1
• Date: Mon, 11 May 2026 10:51:02 GMT
• Title: Mutual Enhancement Between Global Tokens and Patch Tokens: From Theory to Practice
• Abstract summary: TaTok is a Theoretically grounded adaptive image Tokenization framework.<n>We introduce global tokens that model mutual information across patch tokens.<n>Experiments confirm TaTok's state-of-the-art performance, delivering a 1.3x gFID improvement and 8.7x inference speedup.
• Score: 32.91418593188147
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Accurate and effective discrete image tokenization is crucial for long image sequence processing. However, current methods rigidly compress all content at a fixed rate, ignoring the variable information density of images and leading to either redundancy or information loss. Inspired by information entropy, we propose TaTok, a Theoretically grounded adaptive image Tokenization framework. We rigorously identify two key drawbacks in existing methods: information insufficiency when reconstructing images with patch tokens alone, and information redundancy among patch tokens. To address these, we introduce global tokens that model mutual information across patch tokens, and a Dynamic Token Filtering (DTF) algorithm based on cumulative conditional entropy to eliminate redundancy. Experiments confirm TaTok's state-of-the-art performance, delivering a 1.3x gFID improvement and 8.7x inference speedup. By allocating tokens according to information richness, TaTok enables more compressed yet accurate image tokenization, offering valuable insights for future research.

Related papers

• Improving Flexible Image Tokenizers for Autoregressive Image Generation [53.238708824055664]
  textbfReToK is a flexible tokenizer with underlineRedundant underlineToken Padding and Hierarchical Semantic Regularization.<n>Our method achieves superior generation performance compared with both flexible and fixed-length tokenizers.
  arXiv  Detail & Related papers  (2026-01-04T14:11:45Z)
• InfoTok: Adaptive Discrete Video Tokenizer via Information-Theoretic Compression [114.03378443007074]
  Current tokenizers rigidly compress all content at a fixed rate, leading to redundancy or information loss.<n>This paper introduces InfoTok, a principled framework for adaptive video tokenization.<n>We develop a transformer-based adaptive compressor that enables adaptive tokenization.
  arXiv  Detail & Related papers  (2025-12-18T17:13:59Z)
• Hita: Holistic Tokenizer for Autoregressive Image Generation [56.81871174745175]
  We introduce textitHita, a novel image tokenizer for autoregressive (AR) image generation.<n>It introduces a holistic-to-local tokenization scheme with learnable holistic queries and local patch tokens.
  arXiv  Detail & Related papers  (2025-07-03T06:44:26Z)
• One-D-Piece: Image Tokenizer Meets Quality-Controllable Compression [1.7942265700058988]
  We introduce One-D-Piece, a discrete image tokenizer for variable-length tokenization.<n>Tail Token Drop is a regularization mechanism named "Tail Token Drop" into discrete one-dimensional image tokenizers.<n>We evaluate our tokenizer across multiple reconstruction quality metrics and find that it delivers significantly better perceptual quality than existing quality-controllable compression methods.
  arXiv  Detail & Related papers  (2025-01-17T09:29:33Z)
• Adaptive Length Image Tokenization via Recurrent Allocation [81.10081670396956]
  Current vision systems assign fixed-length representations to images, regardless of the information content. Inspired by this, we propose an approach to learn variable-length token representations for 2D images.
  arXiv  Detail & Related papers  (2024-11-04T18:58:01Z)
• Token-level Correlation-guided Compression for Efficient Multimodal Document Understanding [54.532578213126065]
  Most document understanding methods preserve all tokens within sub-images and treat them equally. This neglects their different informativeness and leads to a significant increase in the number of image tokens. We propose Token-level Correlation-guided Compression, a parameter-free and plug-and-play methodology to optimize token processing.
  arXiv  Detail & Related papers  (2024-07-19T16:11:15Z)
• LeMeViT: Efficient Vision Transformer with Learnable Meta Tokens for Remote Sensing Image Interpretation [37.72775203647514]
  This paper proposes to use learnable meta tokens to formulate sparse tokens, which effectively learn key information and improve inference speed. By employing Dual Cross-Attention (DCA) in the early stages with dense visual tokens, we obtain the hierarchical architecture LeMeViT with various sizes. Experimental results in classification and dense prediction tasks show that LeMeViT has a significant $1.7 times$ speedup, fewer parameters, and competitive performance compared to the baseline models.
  arXiv  Detail & Related papers  (2024-05-16T03:26:06Z)
• Token-Label Alignment for Vision Transformers [93.58540411138164]
  Data mixing strategies (e.g., CutMix) have shown the ability to greatly improve the performance of convolutional neural networks (CNNs) We identify a token fluctuation phenomenon that has suppressed the potential of data mixing strategies. We propose a token-label alignment (TL-Align) method to trace the correspondence between transformed tokens and the original tokens to maintain a label for each token.
  arXiv  Detail & Related papers  (2022-10-12T17:54:32Z)
• Not All Patches are What You Need: Expediting Vision Transformers via Token Reorganizations [37.11387992603467]
  Vision Transformers (ViTs) take all the image patches as tokens and construct multi-head self-attention (MHSA) among them. Examples include tokens containing semantically meaningless or distractive image backgrounds. We propose to reorganize image tokens during the feed-forward process of ViT models, which is integrated into ViT during training.
  arXiv  Detail & Related papers  (2022-02-16T00:19:42Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.