Foveation-Guided Dynamic Token Selection for Robust and Efficient Vision Transformers

• URL: http://arxiv.org/abs/2607.09480v1
• Date: Fri, 10 Jul 2026 14:53:16 GMT
• Title: Foveation-Guided Dynamic Token Selection for Robust and Efficient Vision Transformers
• Abstract summary: Foveated Dynamic Transformer (FDT) integrates foveation-guided dynamic token-selection architecture into vision transformer framework.<n>FDT exhibits strong resilience to various types of noise and adversarial attacks, despite not being explicitly trained for such challenges.<n>These attributes position as an HVS-inspired step toward artificial neural networks that combine adaptive computation with improved resilience.
• Score: 33.36466188187489
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The human visual system (HVS) employs foveated sampling and eye movements to achieve efficient perception, conserving both metabolic energy and computational resources. Drawing inspiration from this robustness and adaptability, we introduce the Foveated Dynamic Transformer (FDT), a foveation-guided dynamic token-selection architecture that integrates these mechanisms into a vision transformer framework. The FDT exhibits strong resilience to various types of noise and adversarial attacks, despite not being explicitly trained for such challenges. This inherent robustness is achieved through the use of fixation and foveation modules: the fixation module identifies fixation points to filter out irrelevant information, while the foveation module generates foveated embeddings with multi-scale information. At the 50% fixation-budget setting, FDT achieves higher accuracy than DeiT-S (81.9% vs. 80.9%) while reducing multiply-accumulate operations by 34.57%, highlighting one operating point on its accuracy-efficiency trade-off. These attributes position FDT as an HVS-inspired step toward artificial neural networks that combine adaptive computation with improved resilience.

Related papers

• Deformable Gaussian Occupancy: Decoupling Rigid and Nonrigid Motion with Factorized Distillation [81.44955493339835]
  DeGO is a deformable Gaussian occupancy framework that unifies decoupled Gaussian deformation with factorized 4D foundation-model distillation.<n> Experiments on the Occ3D-NuScenes benchmark demonstrate that our method achieves state-of-the-art performance under weak supervision.
  arXiv  Detail & Related papers  (2026-05-27T15:09:55Z)
• SAFformer:Improving Spiking Transformer via Active Predictive Filtering [6.339831937684366]
  Spiking Neural Networks (SNNs) offer notable advantages in biological plausibility and energy efficiency.<n>We propose SAFformer, a novel Spiking Transformer architecture based on an active predictive filtering paradigm.<n>Experiments show that SAFformer establishes new state-of-the-art performance on CIFAR-10/100 and CIFAR10-DVS.
  arXiv  Detail & Related papers  (2026-05-08T03:17:52Z)
• ToaSt: Token Channel Selection and Structured Pruning for Efficient ViT [14.21482208417138]
  Vision Transformers (ViTs) have achieved remarkable success across various vision tasks, yet their deployment is often hindered by prohibitive computational costs.<n>We propose ToaSt, a decoupled framework applying specialized strategies to distinct ViT components.
  arXiv  Detail & Related papers  (2026-02-17T16:52:13Z)
• Structured Contrastive Learning for Interpretable Latent Representations [2.8870482999983094]
  We propose Structured Contrastive Learning (SCL), a framework that partitions latent space representations into three semantic groups.<n> Experiments on ECG phase invariance and IMU rotation demonstrate superior performance.<n>This work represents a paradigm shift from reactive data augmentation to proactive structural learning, enabling interpretable latent representations in neural networks.
  arXiv  Detail & Related papers  (2025-11-18T21:18:20Z)
• RT-DETRv4: Painlessly Furthering Real-Time Object Detection with Vision Foundation Models [48.91205564876609]
  We propose a cost-effective and highly adaptable distillation framework to enhance lightweight object detectors.<n>Our approach painlessly delivers striking and consistent performance gains across diverse DETR-based models.<n>Our new model family, RT-DETRv4, achieves state-of-the-art results on COCO, attaining AP scores of 49.7/53.5/55.4/57.0 at corresponding speeds of 273/169/124/78 FPS.
  arXiv  Detail & Related papers  (2025-10-29T08:13:17Z)
• Hourglass Tokenizer for Efficient Transformer-Based 3D Human Pose Estimation [73.31524865643709]
  We present a plug-and-play pruning-and-recovering framework, called Hourglass Tokenizer (HoT), for efficient transformer-based 3D pose estimation from videos. Our HoDT begins with pruning pose tokens of redundant frames and ends with recovering full-length tokens, resulting in a few pose tokens in the intermediate transformer blocks. Our method can achieve both high efficiency and estimation accuracy compared to the original VPT models.
  arXiv  Detail & Related papers  (2023-11-20T18:59:51Z)
• DAT++: Spatially Dynamic Vision Transformer with Deformable Attention [87.41016963608067]
  We present Deformable Attention Transformer ( DAT++), a vision backbone efficient and effective for visual recognition. DAT++ achieves state-of-the-art results on various visual recognition benchmarks, with 85.9% ImageNet accuracy, 54.5 and 47.0 MS-COCO instance segmentation mAP, and 51.5 ADE20K semantic segmentation mIoU.
  arXiv  Detail & Related papers  (2023-09-04T08:26:47Z)
• Deeper Insights into ViTs Robustness towards Common Corruptions [82.79764218627558]
  We investigate how CNN-like architectural designs and CNN-based data augmentation strategies impact on ViTs' robustness towards common corruptions. We demonstrate that overlapping patch embedding and convolutional Feed-Forward Network (FFN) boost performance on robustness. We also introduce a novel conditional method enabling input-varied augmentations from two angles.
  arXiv  Detail & Related papers  (2022-04-26T08:22:34Z)
• MIA-Former: Efficient and Robust Vision Transformers via Multi-grained Input-Adaptation [14.866949449862226]
  Vision Transformer (ViT) models are too computationally expensive to be fitted onto real-world resource-constrained devices. We propose a Multi-grained Input-adaptive Vision Transformer framework dubbed MIA-Former that can input-adaptively adjust the structure of ViTs. Experiments and ablation studies validate that the proposed MIA-Former framework can effectively allocate budgets adaptive to the difficulty of input images.
  arXiv  Detail & Related papers  (2021-12-21T22:06:24Z)
• AdaViT: Adaptive Tokens for Efficient Vision Transformer [91.88404546243113]
  We introduce AdaViT, a method that adaptively adjusts the inference cost of vision transformer (ViT) for images of different complexity. AdaViT achieves this by automatically reducing the number of tokens in vision transformers that are processed in the network as inference proceeds.
  arXiv  Detail & Related papers  (2021-12-14T18:56:07Z)
• AdaViT: Adaptive Vision Transformers for Efficient Image Recognition [78.07924262215181]
  We introduce AdaViT, an adaptive framework that learns to derive usage policies on which patches, self-attention heads and transformer blocks to use. Our method obtains more than 2x improvement on efficiency compared to state-of-the-art vision transformers with only 0.8% drop of accuracy.
  arXiv  Detail & Related papers  (2021-11-30T18:57:02Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.