Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection

• URL: http://arxiv.org/abs/2603.13782v1
• Date: Sat, 14 Mar 2026 06:26:11 GMT
• Title: Your Vision-Language-Action Model Already Has Attention Heads For Path Deviation Detection
• Abstract summary: Vision-Language-Action (VLA) models have demonstrated strong potential for predicting semantic actions in navigation tasks.<n>They are fundamentally hindered by visual-reasoning hallucinations that lead to trajectory deviations.<n>We show that monitoring a few attention heads within a frozen VLA model can accurately detect path deviations without incurring additional computational overhead.
• Score: 14.088116748512428
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models have demonstrated strong potential for predicting semantic actions in navigation tasks, demonstrating the ability to reason over complex linguistic instructions and visual contexts. However, they are fundamentally hindered by visual-reasoning hallucinations that lead to trajectory deviations. Addressing this issue has conventionally required training external critic modules or relying on complex uncertainty heuristics. In this work, we discover that monitoring a few attention heads within a frozen VLA model can accurately detect path deviations without incurring additional computational overhead. We refer to these heads, which inherently capture the spatiotemporal causality between historical visual sequences and linguistic instructions, as Navigation Heads. Using these heads, we propose an intuitive, training-free anomaly-detection framework that monitors their signals to detect hallucinations in real time. Surprisingly, among over a thousand attention heads, a combination of just three is sufficient to achieve a 44.6 % deviation detection rate with a low false-positive rate of 11.7 %. Furthermore, upon detecting a deviation, we bypass the heavy VLA model and trigger a lightweight Reinforcement Learning (RL) policy to safely execute a shortest-path rollback. By integrating this entire detection-to-recovery pipeline onto a physical robot, we demonstrate its practical robustness. All source code will be publicly available.

Related papers

• GhostPoint: Self-Supervised Representation Learning by Hallucinating Occluded LiDAR Structure [58.617405464262646]
  3D object detection from LiDAR point clouds is a core problem in autonomous driving.<n>Recent advances in self-supervised learning (SSL) enable scalable pretraining and transfers well to per-point tasks.<n>We analyze recent SSL methods and find that most objectives are defined only on measured LiDAR returns from visible surfaces.<n>We propose GhostPoint, an SSL framework that hallucinates latent features in local neighborhoods around discovered instances.
  arXiv  Detail & Related papers  (2026-08-14T16:10:35Z)
• Role-Break in Attention Heads: Understanding and Detecting Hallucinations in VLMs [49.743008490747094]
  Vision-Language Models (VLMs) remain prone to hallucinations, producing content that is inconsistent with or unsupported by the input image.<n>Under a unified head-level view, we find that hallucination-induced changes manifest as localized deviations from each head's faithful contextual behavior.<n> Detailed analysis reveals that these deviations are systematically organized across attention heads, contextual sources, and deviation directions, and that the resulting signal is linearly readable once head identity is preserved.
  arXiv  Detail & Related papers  (2026-07-31T13:33:51Z)
• Would You Walk to the Car Wash? Revealing the Salience Bias of Large Language Models in Commonsense Reasoning [19.177595059532557]
  We show that models become easily hijacked by useless explicit distractors, leading them to ignore the implicit physical or commonsense prerequisites of a task.<n>A critical open question is whether this failure reflects a genuine gap in commonsense knowledge or merely its suppression under misleading task framing.<n>Our findings relocate the bottleneck of commonsense reasoning failures from model competence to elicitation, and we release SaliTrap as a testbed for this blind spot.
  arXiv  Detail & Related papers  (2026-07-30T16:30:08Z)
• Scalable Token-Level Hallucination Detection in Large Language Models [63.3426544914783]
  Internal hallucinations are difficult to detect in reasoning-intensive tasks.<n>We propose TokenHD, a holistic pipeline for training token-level hallucination detectors.
  arXiv  Detail & Related papers  (2026-05-12T16:47:40Z)
• Mechanisms of Introspective Awareness [4.6752132340946835]
  Recent work shows that LLMs can sometimes detect when steering vectors are injected into their residual stream and identify the injected concept.<n>But what mechanisms underlie this capability, and do they reflect genuine introspective circuitry or more shallow awarenesss?<n>We investigate these questions in open-source models and establish three main findings.
  arXiv  Detail & Related papers  (2026-03-22T20:45:34Z)
• Mitigating Cognitive Inertia in Large Reasoning Models via Latent Spike Steering [12.332146893333949]
  Large Reasoning Models (LRMs) have achieved remarkable performance by scaling test-time compute.<n>LRMs frequently suffer from Cognitive Inertia, a failure pattern manifesting as either overthinking (inertia of motion) or rigidity (inertia of direction)
  arXiv  Detail & Related papers  (2026-01-30T02:47:12Z)
• Vision-Language Introspection: Mitigating Overconfident Hallucinations in MLLMs via Interpretable Bi-Causal Steering [33.3588824921581]
  We introduce Vision-Language Introspection (VLI), a training-free inference framework that simulates a metacognitive self-correction process.<n>VLI achieves state-of-the-art performance on advanced models, reducing object hallucination rates by 12.67% on MMHal-Bench and improving accuracy by 5.8% on POPE.
  arXiv  Detail & Related papers  (2026-01-08T17:49:13Z)
• VOGUE: Guiding Exploration with Visual Uncertainty Improves Multimodal Reasoning [62.09195763860549]
  Reinforcement learning with verifiable rewards (RLVR) improves reasoning in large language models (LLMs) but struggles with exploration.<n>We introduce $textbfVOGUE (Visual Uncertainty Guided Exploration)$, a novel method that shifts exploration from the output (text) to the input (visual) space.<n>Our work shows that grounding exploration in the inherent uncertainty of visual inputs is an effective strategy for improving multimodal reasoning.
  arXiv  Detail & Related papers  (2025-10-01T20:32:08Z)
• INSIGHT: INference-time Sequence Introspection for Generating Help Triggers in Vision-Language-Action Models [2.509305596181814]
  Recent Vision-Language-Action (VLA) models show strong generalization capabilities, yet they lack introspective mechanisms for anticipating failures and requesting help from a human supervisor.<n>We present textbfINSIGHT, a learning framework for leveraging token-level uncertainty signals to predict when a VLA should request help.
  arXiv  Detail & Related papers  (2025-10-01T19:22:48Z)
• Anomalous Decision Discovery using Inverse Reinforcement Learning [3.3675535571071746]
  Anomaly detection plays a critical role in Autonomous Vehicles (AVs) by identifying unusual behaviors through perception systems.<n>Current approaches, which often rely on predefined thresholds or supervised learning paradigms, exhibit reduced efficacy when confronted with unseen scenarios.<n>We present Trajectory-Reward Guided Adaptive Pre-training (TRAP), a novel IRL framework for anomaly detection.
  arXiv  Detail & Related papers  (2025-07-06T17:01:02Z)
• Visual hallucination detection in large vision-language models via evidential conflict [24.465497252040294]
  Dempster-Shafer theory (DST)-based visual hallucination detection method for LVLMs through uncertainty estimation.<n>We propose to the best of our knowledge, the first Dempster-Shafer theory (DST)-based visual hallucination detection method for LVLMs through uncertainty estimation.
  arXiv  Detail & Related papers  (2025-06-24T11:03:10Z)
• Uncertainty-Aware Attention Heads: Efficient Unsupervised Uncertainty Quantification for LLMs [129.79394562739705]
  Large language models (LLMs) exhibit impressive fluency, but often produce critical errors known as "hallucinations"<n>We propose RAUQ (Recurrent Attention-based Uncertainty Quantification), an unsupervised approach that leverages intrinsic attention patterns in transformers to detect hallucinations efficiently.<n> Experiments across 4 LLMs and 12 question answering, summarization, and translation tasks demonstrate that RAUQ yields excellent results.
  arXiv  Detail & Related papers  (2025-05-26T14:28:37Z)
• Attention Tracker: Detecting Prompt Injection Attacks in LLMs [62.247841717696765]
  Large Language Models (LLMs) have revolutionized various domains but remain vulnerable to prompt injection attacks.<n>We introduce the concept of the distraction effect, where specific attention heads shift focus from the original instruction to the injected instruction.<n>We propose Attention Tracker, a training-free detection method that tracks attention patterns on instruction to detect prompt injection attacks.
  arXiv  Detail & Related papers  (2024-11-01T04:05:59Z)
• Detecting Invisible People [58.49425715635312]
  We re-purpose tracking benchmarks and propose new metrics for the task of detecting invisible objects. We demonstrate that current detection and tracking systems perform dramatically worse on this task. Second, we build dynamic models that explicitly reason in 3D, making use of observations produced by state-of-the-art monocular depth estimation networks.
  arXiv  Detail & Related papers  (2020-12-15T16:54:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.