Observing and Controlling Features in Vision-Language-Action Models

• URL: http://arxiv.org/abs/2603.05487v1
• Date: Thu, 05 Mar 2026 18:53:50 GMT
• Title: Observing and Controlling Features in Vision-Language-Action Models
• Abstract summary: Vision-Language-Action Models (VLAs) have shown remarkable progress towards embodied intelligence.<n>We introduce and analyze two main concepts: feature-observability and feature-controllability.<n>Our results show that targeted, lightweight interventions can reliably steer a robot's behavior.
• Score: 20.193036403223903
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action Models (VLAs) have shown remarkable progress towards embodied intelligence. While their architecture partially resembles that of Large Language Models (LLMs), VLAs exhibit higher complexity due to their multi-modal inputs/outputs and often hybrid nature of transformer and diffusion heads. This is part of the reason why insights from mechanistic interpretability in LLMs, which explain how the internal model representations relate to their output behavior, do not trivially transfer to VLA counterparts. In this work, we propose to close this gap by introducing and analyzing two main concepts: feature-observability and feature-controllability. In particular, we first study features that are linearly encoded in representation space, and show how they can be observed by means of a linear classifier. Then, we use a minimal linear intervention grounded in optimal control to accurately place internal representations and steer the VLA's output towards a desired region. Our results show that targeted, lightweight interventions can reliably steer a robot's behavior while preserving closed-loop capabilities. We demonstrate on different VLA architectures ($π_{0.5}$ and OpenVLA) through simulation experiments that VLAs possess interpretable internal structure amenable to online adaptation without fine-tuning, enabling real-time alignment with user preferences and task requirements.

Related papers

• DiMaS: Distribution Matching for Steering Vision-Language-Action Models [41.9662194189542]
  Flow-matching vision-language-action (VLA) models have emerged as powerful policies for robotic manipulation.<n> behavioral control is the ability to govern how a robot performs a task by intervening on its internal representations.<n>We propose DiMaS, a Distribution-Matching Steering strategy tailored to flow-matching VLAs.
  arXiv  Detail & Related papers  (2026-07-15T18:39:13Z)
• Decoupling the Declarative from the Procedural in Vision-Language-Action Models [65.22639791239932]
  Policy trained to clone a behavior from object-specific demonstrations must generalize beyond that object.<n>We propose w$2$VLA, a new Vision-Language-Action model with restructured information flow.<n>Unlike popular, state-of-the-art VLAs, we show that our modular approach successfully decouples knowledge representations.
  arXiv  Detail & Related papers  (2026-06-19T14:43:58Z)
• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• GuidedVLA: Specifying Task-Relevant Factors via Plug-and-Play Action Attention Specialization [101.37117235471709]
  We introduce GuidedVLA, a framework designed to guide the action generation to focus on task-relevant factors.<n>Our core insight is to treat the action decoder not as a monolithic learner, but as an assembly of functional components.<n>Our results suggest that explicitly guiding action-decoder learning is a promising direction for building more robust and general VLA models.
  arXiv  Detail & Related papers  (2026-05-12T16:38:40Z)
• OneDrive: Unified Multi-Paradigm Driving with Vision-Language-Action Models [69.2503510410147]
  We present a unified autonomous driving framework built upon a pretrained VLM.<n>We demonstrate that pretrained VLM attention exhibits strong transferability beyond pure language modeling.<n>Experiments on end-to-end autonomous driving benchmarks demonstrate state-of-the-art performance.
  arXiv  Detail & Related papers  (2026-04-20T07:50:00Z)
• On Integrating Resilience and Human Oversight into LLM-Assisted Modeling Workflows for Digital Twins [0.0]
  We present three critical design principles for integrating resilience and oversight into Digital Twins.<n>A key contribution is detailed characterization of LLM-induced errors across model descriptions of varying detail and complexity.
  arXiv  Detail & Related papers  (2026-03-26T20:38:03Z)
• Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models [16.73630874846666]
  Visual/geometrical and textual representations of spatial structure must be combined at some point in Vision Language Models (VLMs)<n>We show thatVLMs encode object locations by linearly binding itspatialtemporal IDs to textual activations, then perform reasoning via language tokens.<n>We extend our analysis to video VLMs and identify an analogous linear temporal ID mechanism.
  arXiv  Detail & Related papers  (2026-01-18T23:48:38Z)
• Vision-Language-Action Models for Autonomous Driving: Past, Present, and Future [125.92052530850425]
  Vision-Language-Action (VLA) frameworks integrate perception with language-grounded decision making.<n>VLA frameworks offer a pathway toward more interpretable, generalizable, and human-aligned driving policies.<n>This work aims to establish a coherent foundation for advancing human-compatible autonomous driving systems.
  arXiv  Detail & Related papers  (2025-12-18T16:57:44Z)
• Mechanistic interpretability for steering vision-language-action models [0.23371356738437823]
  Vision-Language-Action (VLA) models are a promising path to realizing generalist embodied agents.<n>We introduce the first framework for interpreting and steering VLAs via their internal representations.<n>We introduce a general-purpose activation steering method that modulates behavior in real time, without fine-tuning, reward signals, or environment interaction.
  arXiv  Detail & Related papers  (2025-08-30T03:01:57Z)
• SP-VLA: A Joint Model Scheduling and Token Pruning Approach for VLA Model Acceleration [70.72227437717467]
  Vision-Language-Action (VLA) models have attracted increasing attention for their strong control capabilities.<n>Their high computational cost and low execution frequency hinder their suitability for real-time tasks such as robotic manipulation and autonomous navigation.<n>We propose SP-VLA, a unified framework that accelerates VLA models by jointly scheduling models and pruning tokens.
  arXiv  Detail & Related papers  (2025-06-15T05:04:17Z)
• How Visual Representations Map to Language Feature Space in Multimodal LLMs [9.880509106657009]
  We study the mechanism by which vision-language models (VLMs) achieve alignment of visual and linguistic representations.<n>By keeping the language model frozen, we ensure it maintains its original language representations without adaptation to visual data.<n>We reveal the layer-wise progression through which visual representations gradually align with language feature representations, converging in middle-to-later layers.
  arXiv  Detail & Related papers  (2025-06-13T17:34:05Z)
• Sparse Autoencoders Learn Monosemantic Features in Vision-Language Models [50.587868616659826]
  We introduce a comprehensive framework for evaluating monosemanticity at the neuron-level in vision representations.<n>Our experimental results reveal that SAEs trained on Vision-Language Models significantly enhance the monosemanticity of individual neurons.
  arXiv  Detail & Related papers  (2025-04-03T17:58:35Z)
• CoT-VLA: Visual Chain-of-Thought Reasoning for Vision-Language-Action Models [89.44024245194315]
  We introduce a method that incorporates explicit visual chain-of-thought (CoT) reasoning into vision-language-action models (VLAs)<n>We introduce CoT-VLA, a state-of-the-art 7B VLA that can understand and generate visual and action tokens.<n>Our experimental results demonstrate that CoT-VLA achieves strong performance, outperforming the state-of-the-art VLA model by 17% in real-world manipulation tasks and 6% in simulation benchmarks.
  arXiv  Detail & Related papers  (2025-03-27T22:23:04Z)
• Integrating Frequency-Domain Representations with Low-Rank Adaptation in Vision-Language Models [0.6715525121432597]
  This research presents a novel vision language model (VLM) framework to enhance feature extraction, scalability, and efficiency.<n>We evaluate the proposed model on caption generation and Visual Question Answering (VQA) tasks using benchmark datasets with varying levels of Gaussian noise.<n>Our model provides more detailed and contextually relevant responses, particularly for real-world images captured by a RealSense camera mounted on an Unmanned Ground Vehicle (UGV)
  arXiv  Detail & Related papers  (2025-03-08T01:22:10Z)
• PEVL: Position-enhanced Pre-training and Prompt Tuning for Vision-language Models [127.17675443137064]
  We introduce PEVL, which enhances the pre-training and prompt tuning of vision-language models with explicit object position modeling. PEVL reformulates discretized object positions and language in a unified language modeling framework. We show that PEVL enables state-of-the-art performance on position-sensitive tasks such as referring expression comprehension and phrase grounding.
  arXiv  Detail & Related papers  (2022-05-23T10:17:53Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.