Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking

• URL: http://arxiv.org/abs/2608.03231v1
• Date: Tue, 04 Aug 2026 07:03:05 GMT
• Title: Structure-Aware Robust Fine-Tuning: Defending Vision-Language-Action Robots Against Physical Attention Hijacking
• Abstract summary: Vision-Language-Action (VLA) policies promise general robotic manipulation, but their robustness against physical-world attacks remains fragile.<n>We show that physically realizable adversarial patches can reliably induce failures by triggering a mechanism we call policy-critical action-to-vision attention hijacking.<n>We introduce Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead defense that fine-tunes only the visual encoder using feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions.
• Score: 34.12193936594126
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) policies promise general robotic manipulation, but their robustness against physical-world attacks remains fragile. In particular, we show that physically realizable adversarial patches can reliably induce failures by triggering a mechanism we call policy-critical action-to-vision attention hijacking, where action-conditioned attention is diverted from task-relevant regions to a localized patch. To demonstrate the threat, we propose Attention-Guided Semantic Disruption (AGSD), an Expectation-over-Transformation (EOT) optimized printable patch that jointly (i) concentrates action-to-vision attention on the patch and (ii) disrupts vision-language semantic alignment, yielding strong cross-task and cross-architecture transfer. To mitigate such attacks, we introduce Structure-Aware Robust Fine-Tuning (SARF), a zero-inference-overhead defense that fine-tunes only the visual encoder using feature anchoring, policy-critical attention correction, and language-guided geometric consistency restricted to semantically relevant regions. On LIBERO, SARF reduces OpenVLA's failure rate under AGSD from 100% to 14.2%-56.8% (28.6% average) across suites while preserving clean performance, and on a real PiPER manipulator it improves average success under AGSD from 23.0% to 65.0%. These results highlight mechanism-level robustness as a practical path to securing VLA robots against physical attention hijacking.

Related papers

• VLAGuard: A Framework for Evaluating and Mitigating Physical Attention Hijacking in Vision-Language-Action Robots within Wireless Sensor Networks [1.9680580983094325]
  We present a framework to assess and mitigate a critical vulnerability: policy-critical action-to-vision attention hijacking.<n>We propose AttentionProtective Fine-Tuning (APFT), a defense that stabilizestemporal attention and enforces geometric consistency with zero inference overhead.
  arXiv  Detail & Related papers  (2026-08-02T06:06:28Z)
• Partially Observable Adversarial Patch Attacks on Vision-Language-Action Models in Robotics [21.834006622805678]
  Vision-language-action (VLA) models are gaining attention in robotics, yet their robustness to adversarial attacks remains largely unexplored.<n>We formulate a partially observable threat model, where an adversary can exploit only a short prefix of the trajectory to generate a fixed patch applied to all subsequent frames.<n>First, we localize the patch using the model's attention maps to identify visually critical regions that correspond to the full instruction.<n>Then, we optimize the patch to disrupt the semantic grounding of target objects and increase the curvature of action trajectories.
  arXiv  Detail & Related papers  (2026-06-02T12:19:28Z)
• VLA-Hijack: A Transferable Patch Attack against Vision-Language-Action Models via Visual Proprioception Hijacking [49.51251398232003]
  VLA-Hijack is a unified adversarial framework for Vision-Language-Action models.<n>It exploits a fundamental vulnerability by exploiting a shared visual self-localization process.<n>It achieves superior optimization efficiency in white-box settings and sets a new SOTA for cross-architecture and cross-domain black-box transferability.
  arXiv  Detail & Related papers  (2026-05-27T07:38:16Z)
• ATAAT: Adaptive Threat-Aware Adversarial Tuning Framework against Backdoor Attacks on Vision-Language-Action Models [11.93174278125059]
  This study investigates backdoor attacks targeting the visual pathway.<n>We identify a core obstacle causing the failure of traditional attack paradigms: "Gradient Interference"<n>We propose an Adaptive Threat-Aware Adrial Tuning framework.
  arXiv  Detail & Related papers  (2026-05-09T02:15:10Z)
• Uncovering Linguistic Fragility in Vision-Language-Action Models via Diversity-Aware Red Teaming [64.48633529149579]
  We propose a novel framework to expose the vulnerabilities of Vision-Language-Action (VLA) models against linguistic variations.<n>Our method reduces the average task success rate from 93.33% to 5.85%, demonstrating a scalable approach to stress-testing VLA agents.
  arXiv  Detail & Related papers  (2026-04-07T08:43:36Z)
• When Robots Obey the Patch: Universal Transferable Patch Attacks on Vision-Language-Action Models [81.7618160628979]
  Vision-Language-Action (VLA) models are vulnerable to adversarial attacks, yet universal and transferable attacks remain underexplored.<n>We introduce UPA-RFAS (Universal Patch Attack via Robust Feature, Attention, and Semantics), a unified framework that learns a single physical patch in a shared feature space.<n> Experiments across diverse VLA models, manipulation suites, and physical executions show that UPA-RFAS consistently transfers across models, tasks, and viewpoints.
  arXiv  Detail & Related papers  (2025-11-26T09:16:32Z)
• AttackVLA: Benchmarking Adversarial and Backdoor Attacks on Vision-Language-Action Models [60.39655329875822]
  Vision-Language-Action (VLA) models enable robots to interpret natural-language instructions and perform diverse tasks.<n>Despite growing interest in attacking such models, the effectiveness of existing techniques remains unclear.<n>We propose AttackVLA, a unified framework that aligns with the VLA development lifecycle.
  arXiv  Detail & Related papers  (2025-11-15T10:30:46Z)
• Exploring the Adversarial Vulnerabilities of Vision-Language-Action Models in Robotics [68.36528819227641]
  This paper systematically evaluates the robustness of Vision-Language-Action (VLA) models.<n>We introduce two untargeted attack objectives that leverage spatial foundations to destabilize robotic actions, and a targeted attack objective that manipulates the robotic trajectory.<n>We design an adversarial patch generation approach that places a small, colorful patch within the camera's view, effectively executing the attack in both digital and physical environments.
  arXiv  Detail & Related papers  (2024-11-18T01:52:20Z)
• Towards Robust Semantic Segmentation against Patch-based Attack via Attention Refinement [68.31147013783387]
  We observe that the attention mechanism is vulnerable to patch-based adversarial attacks. In this paper, we propose a Robust Attention Mechanism (RAM) to improve the robustness of the semantic segmentation model.
  arXiv  Detail & Related papers  (2024-01-03T13:58:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.