Steering Autoregressive Vision-Language-Action Policies via Action Token Intervention

• URL: http://arxiv.org/abs/2606.15021v1
• Date: Fri, 12 Jun 2026 23:37:01 GMT
• Title: Steering Autoregressive Vision-Language-Action Policies via Action Token Intervention
• Abstract summary: Token Steering (TS) is a method for dynamically steering trajectories generated by an autoregressive vision-language-action (VLA) model.<n>TS injects low-dimensional user inputs into the model's native action-token representation.<n>Because TS operates entirely at inference time, it requires no additional training or finetuning.
• Score: 4.130353156902536
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: We present Token Steering (TS), a method for dynamically steering trajectories generated by an autoregressive vision-language-action (VLA) model through direct intervention in the action-token space. TS injects low-dimensional user inputs into the model's native action-token representation, allowing users to influence trajectory generation without modifying the underlying vision-language model (VLM) architecture. Because TS operates entirely at inference time, it requires no additional training or finetuning. User inputs guide rather than override the pretrained policy, allowing users to influence robot actions while preserving the dexterity, smoothness, and task priors learned by the VLA. We evaluate TS on two household manipulation tasks -- drawer closing after object placement and state-aware object swapping -- and improve success rates from 10.0% to 72.5% and from 16.7% to 93.8%, respectively. By enabling lightweight, intuitive steering over robot foundation models, our interface has the potential to improve human-robot interaction in consumer environments and broaden accessibility for individuals with limited physical control. Project website: https://jasontchan.github.io/token-steering/ .

Related papers

• Assistron: Bayesian Shared Autonomy with Off-the-shelf Vision-Language-Action Models [6.557127548028966]
  We propose Assistron, a shared autonomy model that leverages Vision-Language-Action (VLA) models to assist the user in daily activities.<n>Our approach is grounded in two core principles: (1)minimizing human cognitive and physical effort by leveraging VLA-driven autonomy for macro-movements, and (2)prioritizing human intervention specifically at critical failure points.
  arXiv  Detail & Related papers  (2026-06-22T10:47:12Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• Wall-OSS-0.5 Technical Report [13.983843529533113]
  We present Wall-OSS-0.5, an open-source 4B VLA built upon a 3B VLM backbone augmented with action-generation components.<n>The model is pretrained across more than 20 embodiments, processing over one million robot trajectories per epoch.<n>It achieves non-trivial zero-shot real-robot behavior, completing several tasks, including a held-out deformable manipulation task, at high task progress on a 17-task suite.
  arXiv  Detail & Related papers  (2026-05-29T06:04:03Z)
• On-the-Fly VLA Adaptation via Test-Time Reinforcement Learning [61.38273866396522]
  Vision-Language-Action models have emerged as a powerful paradigm for general-purpose robot learning.<n>Current methods remain unsuitable for challenging simulated- or physical-world deployments.<n>We introduce a Test-Time Reinforcement Learning for VLAs framework that enables on-the-fly policy adaptation during inference.
  arXiv  Detail & Related papers  (2026-01-11T01:51:30Z)
• VITA-VLA: Efficiently Teaching Vision-Language Models to Act via Action Expert Distillation [76.13140980997508]
  Vision-Language Action (VLA) models significantly advance robotic manipulation by leveraging the strong perception capabilities of pretrained vision-language models (VLMs)<n>We propose a simple yet effective distillation-based framework that equips VLMs with action-execution capability by transferring knowledge from pretrained small action models.<n>In real-world experiments across five manipulation tasks, our method consistently outperforms the teacher model, achieving 82.0% success rate (17% improvement)
  arXiv  Detail & Related papers  (2025-10-10T17:59:56Z)
• VISOR: Visual Input-based Steering for Output Redirection in Vision-Language Models [1.4262180230002854]
  VISOR (Visual Input-based Steering for Output Redirection) is a novel method that achieves sophisticated behavioral control through optimized visual inputs alone.<n>We validate VISOR on LLaVA-1.5-7B across three critical alignment tasks: refusal, sycophancy and survival instinct.<n>VISOR provides robust bidirectional control while maintaining 99.9% performance on 14,000 unrelated MMLU tasks.
  arXiv  Detail & Related papers  (2025-08-11T23:25:16Z)
• FAST: Efficient Action Tokenization for Vision-Language-Action Models [98.15494168962563]
  We propose a new compression-based tokenization scheme for robot actions, based on the discrete cosine transform.<n>Based on FAST, we release FAST+, a universal robot action tokenizer, trained on 1M real robot action trajectories.
  arXiv  Detail & Related papers  (2025-01-16T18:57:04Z)
• Latent Action Pretraining from Videos [156.88613023078778]
  We introduce Latent Action Pretraining for general Action models (LAPA)<n>LAPA is an unsupervised method for pretraining Vision-Language-Action (VLA) models without ground-truth robot action labels.<n>We propose a method to learn from internet-scale videos that do not have robot action labels.
  arXiv  Detail & Related papers  (2024-10-15T16:28:09Z)
• Run-time Observation Interventions Make Vision-Language-Action Models More Visually Robust [9.647148940880381]
  Vision-language-action (VLA) models trained on large-scale internet data and robot demonstrations have the potential to serve as generalist robot policies. We introduce Bring Your Own VLA (BYOVLA): a run-time intervention scheme that dynamically identifies regions of the input image that the model is sensitive to. We show that BYOVLA enables state-of-the-art VLA models to nearly retain their nominal performance in the presence of distractor objects and backgrounds.
  arXiv  Detail & Related papers  (2024-10-02T19:29:24Z)
• Robotic Control via Embodied Chain-of-Thought Reasoning [86.6680905262442]
  Key limitation of learned robot control policies is their inability to generalize outside their training data.<n>Recent works on vision-language-action models (VLAs) have shown that the use of large, internet pre-trained vision-language models can substantially improve their robustness and generalization ability.<n>We introduce Embodied Chain-of-Thought Reasoning (ECoT) for VLAs, in which we train VLAs to perform multiple steps of reasoning about plans, sub-tasks, motions, and visually grounded features before predicting the robot action.
  arXiv  Detail & Related papers  (2024-07-11T17:31:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.