VAT: Vision Action Transformer by Unlocking Full Representation of ViT

• URL: http://arxiv.org/abs/2512.06013v1
• Date: Wed, 03 Dec 2025 10:50:40 GMT
• Title: VAT: Vision Action Transformer by Unlocking Full Representation of ViT
• Authors: Wenhao Li, Chengwei Ma, Weixin Mao,
• Abstract summary: Vision Transformers (ViTs) are standard for visual perception, yet most methods discard valuable information by using only the final layer's features.<n>We argue this provides an insufficient representation and propose the Vision Action Transformer (VAT)<n>VAT processes specialized action tokens with visual features across all transformer layers, enabling a deep and progressive fusion of perception and action generation.
• Score: 10.192713461564606
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In robot learning, Vision Transformers (ViTs) are standard for visual perception, yet most methods discard valuable information by using only the final layer's features. We argue this provides an insufficient representation and propose the Vision Action Transformer (VAT), a novel architecture that is extended from ViT and unlocks the full feature hierarchy of ViT. VAT processes specialized action tokens with visual features across all transformer layers, enabling a deep and progressive fusion of perception and action generation. On a suite of simulated manipulation tasks, VAT achieves a 98.15\% average success rate across four LIBERO benchmarks, establishing a new state-of-the-art by outperforming prior methods like OpenVLA-OFT. Our work presents not only a powerful model for imitation learning but also demonstrates the critical importance of leveraging the complete ''representation trajectory'' of vision models to advance robotic policy. The GitHub URL for the project code is https://github.com/sellerbubble/VAT.

Related papers

• Recasting Generic Pretrained Vision Transformers As Object-Centric Scene Encoders For Manipulation Policies [23.378072284295005]
  SOFT is a wrapper around pre-trained vision transformer (PVT) models. Rather than construct representations out of only the final layer activations, SOFT individuates and locates object-like entities from PVT attentions. We demonstrate in each case that policies trained on SOFT far outstrip standard PVT representations for manipulation tasks in simulated and real settings.
  arXiv  Detail & Related papers  (2024-05-24T20:20:15Z)
• Hiera: A Hierarchical Vision Transformer without the Bells-and-Whistles [65.54857068975068]
  In this paper, we argue that this additional bulk is unnecessary. By pretraining with a strong visual pretext task (MAE), we can strip out all the bells-and-whistles from a state-of-the-art multi-stage vision transformer. We create Hiera, an extremely simple hierarchical vision transformer that is more accurate than previous models.
  arXiv  Detail & Related papers  (2023-06-01T17:59:58Z)
• What do Vision Transformers Learn? A Visual Exploration [68.50771218442776]
  Vision transformers (ViTs) are quickly becoming the de-facto architecture for computer vision. This paper addresses the obstacles to performing visualizations on ViTs and explores the underlying differences between ViTs and CNNs. We also conduct large-scale visualizations on a range of ViT variants, including DeiT, CoaT, ConViT, PiT, Swin, and Twin.
  arXiv  Detail & Related papers  (2022-12-13T16:55:12Z)
• HiViT: Hierarchical Vision Transformer Meets Masked Image Modeling [126.89573619301953]
  We propose a new design of hierarchical vision transformers named HiViT (short for Hierarchical ViT) HiViT enjoys both high efficiency and good performance in MIM. In running MAE on ImageNet-1K, HiViT-B reports a +0.6% accuracy gain over ViT-B and a 1.9$times$ speed-up over Swin-B.
  arXiv  Detail & Related papers  (2022-05-30T09:34:44Z)
• Auto-scaling Vision Transformers without Training [84.34662535276898]
  We propose As-ViT, an auto-scaling framework for Vision Transformers (ViTs) without training. As-ViT automatically discovers and scales up ViTs in an efficient and principled manner. As a unified framework, As-ViT achieves strong performance on classification and detection.
  arXiv  Detail & Related papers  (2022-02-24T06:30:55Z)
• Scaling Vision Transformers [82.08465256393514]
  We study how Vision Transformers scale and characterize the relationships between error rate, data, and compute. We train a ViT model with two billion parameters, which attains a new state-of-the-art on ImageNet of 90.45% top-1 accuracy. The model also performs well on few-shot learning, for example, attaining 84.86% top-1 accuracy on ImageNet with only 10 examples per class.
  arXiv  Detail & Related papers  (2021-06-08T17:47:39Z)
• On Improving Adversarial Transferability of Vision Transformers [97.17154635766578]
  Vision transformers (ViTs) process input images as sequences of patches via self-attention. We study the adversarial feature space of ViT models and their transferability. We introduce two novel strategies specific to the architecture of ViT models.
  arXiv  Detail & Related papers  (2021-06-08T08:20:38Z)
• CvT: Introducing Convolutions to Vision Transformers [44.74550305869089]
  Convolutional vision Transformer (CvT) improves Vision Transformer (ViT) in performance and efficiency. New architecture introduces convolutions into ViT to yield the best of both designs.
  arXiv  Detail & Related papers  (2021-03-29T17:58:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.