Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers

• URL: http://arxiv.org/abs/2607.02612v1
• Date: Wed, 01 Jul 2026 15:51:25 GMT
• Title: Fusion: A Framework for Unified Sequential Token AdaptatIon in VisiOn TraNsformers
• Abstract summary: Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation.<n>Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early.<n>We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning.
• Score: 2.604903535691346
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Vision Transformers achieve strong image classification accuracy but process all image regions with nearly the same computation, even when many regions are redundant or uninformative. Recent adaptive inference methods reduce this cost by selectively compressing tokens or terminating inference early, but combining these mechanisms often causes unstable intermediate representations and accuracy degradation. We introduce Fusion, a unified adaptive inference framework that coordinates token merging, early exiting, and token pruning through a simple staged design: tokens are merged first, confidence is evaluated next, and pruning is applied only to samples that continue inference. This ordering allows the three mechanisms to operate cooperatively rather than competitively. Fusion further includes lightweight routing modules that adapt compression strength to each input and support inference-time adjustment of the accuracy--latency trade-off without retraining. On ImageNet-1k with DeiT-S, Fusion matches or surpasses state-of-the-art adaptive ViT methods at comparable compute budgets while reducing calibration error by up to $4\times$ and inference energy by $48\%$. Experiments across ImageNet-100, CIFAR-100, and ImageNette with multiple ViT backbones demonstrate consistent transferability without dataset-specific tuning.

Related papers

• Focus When Necessary: Adaptive Routing and Collaborative Grounding for Training-Free Visual Grounding [56.38477453373368]
  LazyMCoT is a dynamic and training-free framework that adaptively allocates visual grounding efforts based on sample difficulty.<n>LazyMCoT rivals training-based approaches by simultaneously improving reasoning accuracy and reducing average inference latency.
  arXiv  Detail & Related papers  (2026-06-15T03:17:44Z)
• AdaMerge: Salience-Aware Adaptive Token Merging for Training-Free Acceleration of Vision Transformers [5.739405014622565]
  AdaMerge is a token-merging framework based on two complementary mechanisms.<n>It consistently outperforms ToMe, PiToMe, and DSM across all FLOPs-matched regimes.
  arXiv  Detail & Related papers  (2026-05-26T05:35:39Z)
• UniCompress: Token Compression for Unified Vision-Language Understanding and Generation [62.943173382496276]
  Unified models aim to support both understanding and generation by encoding images into discrete tokens and processing them alongside text.<n>We propose a unified token compression algorithm that significantly reduces visual token count while preserving performance on both image understanding and generation tasks.
  arXiv  Detail & Related papers  (2026-03-11T21:27:15Z)
• Joint Post-Training Quantization of Vision Transformers with Learned Prompt-Guided Data Generation [5.75627633588113]
  We present a framework for joint quantization of Vision Transformers trained on ImageNet.<n>We achieve state-of-the-art W4A4 and W3A3 accuracies on ImageNet.<n>We also introduce a data-free calibration strategy that synthesizes diverse, label-free samples.
  arXiv  Detail & Related papers  (2026-02-21T15:02:21Z)
• FUSER: Feed-Forward MUltiview 3D Registration Transformer and SE(3)$^N$ Diffusion Refinement [39.19949818461193]
  F is the first feed-forward multiview registration transformer that processes all scans in a unified, compact latent space.<n>F predicts global poses without any pairwise estimation.<n>Experiments on 3DMatch, ScanNet and ArkitScenes demonstrate that our approach achieves the superior registration accuracy and outstanding computational efficiency.
  arXiv  Detail & Related papers  (2025-12-10T07:11:22Z)
• Efficient Token Compression for Vision Transformer with Spatial Information Preserved [59.79302182800274]
  Token compression is essential for reducing the computational and memory requirements of transformer models.<n>We propose an efficient and hardware-compatible token compression method called Prune and Merge.
  arXiv  Detail & Related papers  (2025-03-30T14:23:18Z)
• FUSE: Label-Free Image-Event Joint Monocular Depth Estimation via Frequency-Decoupled Alignment and Degradation-Robust Fusion [92.4205087439928]
  Image-event joint depth estimation methods leverage complementary modalities for robust perception, yet face challenges in generalizability.<n>We propose the Self-supervised Transfer (PST) and the FrequencyDe-coupled Fusion module (FreDF)<n>PST establishes cross-modal knowledge transfer through latent space alignment with image foundation models, effectively mitigating data scarcity.<n>FreDF explicitly decouples high-frequency edge features from low-frequency structural components, resolving modality-specific frequency mismatches.<n>This combined approach enables FUSE to construct a universal image-event that only requires lightweight decoder adaptation for target datasets.
  arXiv  Detail & Related papers  (2025-03-25T15:04:53Z)
• FASTer: Focal Token Acquiring-and-Scaling Transformer for Long-term 3D Object Detection [9.291995455336929]
  We propose a Focal Token Acquring-and-Scaling Transformer (FASTer)<n>FASTer condenses token sequences in an adaptive and lightweight manner.<n>It significantly outperforms other state-of-the-art detectors in both performance and efficiency.
  arXiv  Detail & Related papers  (2025-02-28T03:15:33Z)
• Implicit Grid Convolution for Multi-Scale Image Super-Resolution [6.8410780175245165]
  We propose a multi-scale framework that employs a single encoder in conjunction with Implicit Grid Convolution (IGConv) Our framework achieves comparable performance to existing fixed-scale methods while reducing the training budget and stored parameters three-fold.
  arXiv  Detail & Related papers  (2024-08-19T03:30:15Z)
• Token Fusion: Bridging the Gap between Token Pruning and Token Merging [71.84591084401458]
  Vision Transformers (ViTs) have emerged as powerful backbones in computer vision, outperforming many traditional CNNs. computational overhead, largely attributed to the self-attention mechanism, makes deployment on resource-constrained edge devices challenging. We introduce "Token Fusion" (ToFu), a method that amalgamates the benefits of both token pruning and token merging.
  arXiv  Detail & Related papers  (2023-12-02T04:29:19Z)
• AdaViT: Adaptive Tokens for Efficient Vision Transformer [91.88404546243113]
  We introduce AdaViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. AdaViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
  arXiv  Detail & Related papers  (2021-12-14T18:56:07Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.