Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models

• URL: http://arxiv.org/abs/2608.02197v1
• Date: Mon, 03 Aug 2026 13:18:36 GMT
• Title: Look Where It Matters: Adaptive Visual Refinement for Vision-Language-Action Models
• Abstract summary: We show that visual encoders exhibit attention artifacts previously documented in generic Vision Transformers.<n>We introduce AtVLA, a framework that inserts learnable register tokens into the visual encoder.
• Score: 9.514456645829966
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Visual representations of VLA models remain unreliable for spatially precise robotic manipulation. We uncover that vision encoders in VLAs also exhibit attention artifacts previously documented in generic Vision Transformers, and further show that, in embodied policies, these artifacts are closely associated with spatial perception capabilities acquired during post-training. As the encoder learns task-relevant information such as object location, depth ordering, and local geometry, limited global-token capacity causes part of this information to spill into low-information patch tokens. We introduce AtVLA, a framework that inserts learnable register tokens into the visual encoder. Trained end-to-end using only embodied data and the original action objective, these registers emerge as dedicated carriers of embodied spatial information, while the remaining patch tokens recover clean and spatially faithful attention distributions crucial for precise target localization and fine-grained contact. Clean attention restores reliable localization, but cannot recover geometric details lost in low-resolution observations. AtVLA therefore couples attention rectification with uncertainty-gated local refinement. The action expert samples multiple action chunks and estimates uncertainty from their disagreement; only for uncertain predictions, action-conditioned attention rollout identifies the task-relevant region, which is cropped, re-encoded at high resolution, and appended to the cached prefix for refined action generation. Across LIBERO, SimplerEnv, and a challenging single-view real-world benchmark, AtVLA improves the average LIBERO success rate from 94.2% to 98.4% and real-world success from 46.5% to 69.0%. The cropping is triggered on approximately 30% of replanning steps, resulting in only 1.4-1.6x the total computation of the base model under the representative deployment setting.

Related papers

• From Drop-off to Recovery: A Mechanistic Analysis of Segmentation in MLLMs [50.185593677108436]
  Multimodal Large Language Models (MLLMs) are increasingly applied to pixel-level vision tasks, yet their intrinsic capacity for spatial understanding remains poorly understood.<n>We investigate segmentation capacity through a layerwise linear probing evaluation across the entire MLLM pipeline: vision encoder, adapter, and LLM.
  arXiv  Detail & Related papers  (2026-03-18T00:22:15Z)
• Informative Object-centric Next Best View for Object-aware 3D Gaussian Splatting in Cluttered Scenes [7.75982375074512]
  We introduce an instance-aware Next Best View (NBV) policy that prioritizes underexplored regions by leveraging object features.<n>Specifically, our object-aware 3DGS distills instancelevel information into one-hot object vectors, which are used to compute confidence-weighted information gain.<n>Experiments demonstrate that our NBV policy reduces depth error by up to 77.14% on the synthetic dataset and 34.10% on the real-world GraspNet dataset.
  arXiv  Detail & Related papers  (2026-02-09T04:50:36Z)
• Nüwa: Mending the Spatial Integrity Torn by VLM Token Pruning [82.39668822222386]
  Vision token pruning has proven to be an effective acceleration technique for the efficient Vision Language Model (VLM)<n>We propose $textNwa$, a two-stage token pruning framework that enables efficient feature aggregation while maintaining spatial integrity.<n>Experiments demonstrate that $textNwa$ achieves SOTA performance on multiple VQA benchmarks (from 94% to 95%) and yields substantial improvements on visual grounding tasks (from 7% to 47%)
  arXiv  Detail & Related papers  (2026-02-03T00:51:03Z)
• VVLoc: Prior-free 3-DoF Vehicle Visual Localization [6.151313455860856]
  We propose a unified pipeline that employs a single neural network to concurrently achieve topological and metric vehicle localization using multi-camera system.<n> VVLoc first evaluates the geo-proximity between visual observations, then estimates their relative metric poses using a matching strategy, while also providing a confidence measure.<n>We evaluate VVLoc not only on the publicly available datasets, but also on a more challenging self-collected dataset.
  arXiv  Detail & Related papers  (2026-01-31T16:37:30Z)
• Crane: Context-Guided Prompt Learning and Attention Refinement for Zero-Shot Anomaly Detection [50.343419243749054]
  Anomaly detection is critical in fields such as medical diagnostics and industrial defect detection.<n> CLIP's coarse-grained image-text alignment limits localization and detection performance for fine-grained anomalies.<n>Crane improves the state-of-the-art ZSAD from 2% to 28%, at both image and pixel levels, while remaining competitive in inference speed.
  arXiv  Detail & Related papers  (2025-04-15T10:42:25Z)
• Recognize Any Regions [55.76437190434433]
  RegionSpot integrates position-aware localization knowledge from a localization foundation model with semantic information from a ViL model.<n>Experiments in open-world object recognition show that our RegionSpot achieves significant performance gain over prior alternatives.
  arXiv  Detail & Related papers  (2023-11-02T16:31:49Z)
• Spatial-Aware Token for Weakly Supervised Object Localization [137.0570026552845]
  We propose a task-specific spatial-aware token to condition localization in a weakly supervised manner. Experiments show that the proposed SAT achieves state-of-the-art performance on both CUB-200 and ImageNet, with 98.45% and 73.13% GT-known Loc.
  arXiv  Detail & Related papers  (2023-03-18T15:38:17Z)
• Conditioning Covert Geo-Location (CGL) Detection on Semantic Class Information [5.660207256468971]
  Task for identification of potential hideouts termed Covert Geo-Location (CCGL) detection was proposed by Saha et al. No attempts were made to utilize semantic class information, which is crucial for obscured detection. In this paper, we propose a multitask-learning-based approach to achieve 2 goals - i) extraction of features having semantic class information; ii) robust training of the common encoder, exploiting large standard annotated datasets as training set for the auxiliary task (semantic segmentation).
  arXiv  Detail & Related papers  (2022-11-27T07:21:59Z)
• GliTr: Glimpse Transformers with Spatiotemporal Consistency for Online Action Prediction [26.184988507662535]
  Many online action prediction models observe complete frames to locate and attend to informative subregions in the frames called glimpses. In this paper, we develop Glimpse Transformers (GliTr), which observe only narrow glimpses at all times.
  arXiv  Detail & Related papers  (2022-10-24T21:10:34Z)
• Adaptive Local-Component-aware Graph Convolutional Network for One-shot Skeleton-based Action Recognition [54.23513799338309]
  We present an Adaptive Local-Component-aware Graph Convolutional Network for skeleton-based action recognition. Our method provides a stronger representation than the global embedding and helps our model reach state-of-the-art.
  arXiv  Detail & Related papers  (2022-09-21T02:33:07Z)
• Rethinking Query-Key Pairwise Interactions in Vision Transformers [5.141895475956681]
  We propose key-only attention, which excludes query-key pairwise interactions and uses a compute-efficient saliency-gate to obtain attention weights. We develop a new self-attention model family, LinGlos, which reach state-of-the-art accuracies on the parameter-limited setting of ImageNet classification benchmark.
  arXiv  Detail & Related papers  (2022-07-01T03:36:49Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.