APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies

• URL: http://arxiv.org/abs/2606.12366v1
• Date: Wed, 10 Jun 2026 17:34:25 GMT
• Title: APT: Action Expert Pretraining Improves Instruction Generalization of Vision-Language-Action Policies
• Abstract summary: We propose a two-stage training method emphasizing Action expert PreTraining.<n>In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance.<n>In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior.
• Score: 22.87409999086972
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Vision-Language-Action (VLA) models that couple pretrained Vision-Language Models (VLMs) with continuous action experts have achieved strong manipulation performance, yet generalization to out-of-distribution (OOD) language instructions remains poor. A known challenge is the structural imbalance in VLA data, where language is far less diverse than visual and action content, making policies prone to visual shortcuts. While discrete-action methods mitigate this through vision-language co-training, continuous action experts lack such protection: they start from random initialization and learn entirely from imbalanced data, producing noisy gradients that corrupt the VLM and fail to exploit its language capability. We address this from a Bayesian perspective, factorizing the policy into a language-agnostic Vision-Action (VA) prior and a language-conditioned VLA likelihood, and propose APT, a two-stage training method emphasizing Action expert PreTraining. In Stage 1, the action expert is pretrained as a VA prior on vision-action pairs from a frozen VLM, bypassing the language imbalance. In Stage 2, language tokens are injected through a gated fusion mechanism that integrates VLM features while preserving the learned visuomotor prior. APT applies to mainstream VLA architectures, including the $π$ and GR00T-style architectures. Comprehensive experiments validate that APT achieves consistent gains on unseen instructions and compositional tasks. Project Page: https://xukechun.github.io/papers/APT/

Related papers

• In-Context VLA: Endowing Vision-Language-Action Models with Language via In-Context Post-Training and Agentic Tool Use [18.507118969955666]
  Vision-Language-Action models imitate expert action chunks conditioned on a static image and a fixed instruction.<n>We introduce textbfourmethod, a framework that endows a VLA with language competence through in-context post-training.<n>Our method consistently achieves SOTA results in both performance and efficiency when compared with CoT-based approaches.
  arXiv  Detail & Related papers  (2026-08-06T08:21:02Z)
• LA4VLA: Learning to Act without Seeing via Language-Action Pretraining [51.209359878543005]
  Vision-Language-Action (VLA) models are commonly pretrained on robot demonstrations by jointly mapping visual observations and language instructions to actions.<n>We propose LA4VLA, a language-action pretraining framework that enables policies to acquire language-conditioned action priors without visual observations.
  arXiv  Detail & Related papers  (2026-06-25T17:13:02Z)
• Learning What to Say to Your VLA: Mostly Harmless Vision Language Action Model Steering [19.25748772628753]
  Vision-Language-Action (VLA) models provide a natural language interface to robot control.<n>We propose a framework that interactively searches for language sequences that improve closed-loop VLA task performance.<n>We learn an improvement head that predicts when language steering will improve performance.
  arXiv  Detail & Related papers  (2026-06-10T16:34:49Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• QuoVLA: Quotient Space for Vision-Language-Action Models [51.02329790939691]
  Vision-Language-Action (VLA) models commonly adapt pretrained VLM latents to robot control by mapping visual observations and language instructions to continuous actions.<n>Our textitQuotient Theory for VLA shows that pretrained VLM latents are not action-insufficient but action-sufficient.<n>We propose QuoVLA, a quotient-space framework for VLA that compresses pretrained VLM latents into action-sufficient representations.
  arXiv  Detail & Related papers  (2026-05-24T06:28:53Z)
• Restoring Linguistic Grounding in VLA Models via Train-Free Attention Recalibration [24.562540060971273]
  Vision-Language-Action (VLA) models enable robots to perform manipulation tasks directly from natural language instructions.<n>We reveal a critical failure mode in which VLA policies continue executing visually plausible actions even when the language instruction contradicts the scene.<n>We propose Instruction-Guided Attention Recalibration (IGAR), a train-free inference-time mechanism that rebalances attention to restore the influence of language instructions.
  arXiv  Detail & Related papers  (2026-03-06T08:01:36Z)
• When Vision Overrides Language: Evaluating and Mitigating Counterfactual Failures in VLAs [31.92520697946991]
  Vision-Language-Action models (VLAs) promise to ground language instructions in robot control, yet in practice often fail to faithfully follow language.<n>We show that counterfactual failures are prevalent yet underexplored across state-of-the-art VLAs.<n>We propose Counterfactual Action Guidance (CAG), a simple yet effective dual-branch inference scheme.
  arXiv  Detail & Related papers  (2026-02-19T18:59:20Z)
• Seeing to Act, Prompting to Specify: A Bayesian Factorization of Vision Language Action Policy [59.44168425139687]
  BayesVLA is a Bayesian factorization that decomposes the policy into a visual-action prior, supporting seeing-to-act, and a language-conditioned likelihood, enabling prompt-to-specify.<n>Experiments show superior generalization to unseen instructions, objects, and environments compared to existing methods.
  arXiv  Detail & Related papers  (2025-12-12T01:59:23Z)
• OTTER: A Vision-Language-Action Model with Text-Aware Visual Feature Extraction [95.6266030753644]
  Vision-Language-Action (VLA) models aim to predict robotic actions based on visual observations and language instructions.<n>Existing approaches require fine-tuning pre-trained vision-language models (VLMs) as visual and language features are independently fed into downstream policies.<n>We propose OTTER, a novel VLA architecture that leverages existing alignments through explicit, text-aware visual feature extraction.
  arXiv  Detail & Related papers  (2025-03-05T18:44:48Z)
• VILA: On Pre-training for Visual Language Models [74.08039416548209]
  We study the design options for VLM pre-training through step-by-step controllable comparisons. We build VILA, a Visual Language model family that consistently outperforms the state-of-the-art models.
  arXiv  Detail & Related papers  (2023-12-12T18:58:18Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.