Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs

• URL: http://arxiv.org/abs/2607.02466v1
• Date: Thu, 02 Jul 2026 17:33:37 GMT
• Title: Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs
• Abstract summary: Task-Agnostic Pretraining (TAP) is a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data.<n>On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data.<n>On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%.
• Score: 46.61554174146852
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models are fundamentally bottlenecked by the scarcity of expert demonstrations -- triplets of observations, instructions, and actions that are costly to collect at scale. We argue that this bottleneck stems from conflating two distinct learning objectives: acquiring physical competence (how to move) and acquiring semantic alignment (what to do). Crucially, only the latter requires language supervision. Building on this Decomposition Hypothesis, we propose Task-Agnostic Pretraining (TAP), a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data -- including discarded off-task trajectories and autonomous robot play -- via a self-supervised Inverse Dynamics objective. A lightweight second stage then grounds these priors in language using minimal expert data. On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data, yielding a 10% absolute gain over standard behavior cloning. On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%, demonstrating that task-agnostic pretraining produces robust, transferable physical representations and offers a scalable path forward for Embodied AI.

Related papers

• Fine-Tuning VLAs with Self-Demonstrated Generative Control for Multi-Task Manipulation [13.410971154780588]
  State-of-the-art vision-language-action (VLA) models exhibit strong semantic understanding, instruction following and task behavior.<n>When deployed on new robots, even minor mismatches in hardware configuration relative to pretraining can cause severe performance drops.<n>We propose a self-supervised method that generates online interaction rollouts from the zero-shot VLA as additional training data for finetuning.
  arXiv  Detail & Related papers  (2026-08-19T23:02:07Z)
• StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models [37.45050514108603]
  We present StellaVLA, a framework that adapts at test time by conditioning on a single retrieved demonstration.<n>An automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost.<n>A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone.
  arXiv  Detail & Related papers  (2026-08-12T05:30:53Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• Wall-OSS-0.5 Technical Report [13.983843529533113]
  We present Wall-OSS-0.5, an open-source 4B VLA built upon a 3B VLM backbone augmented with action-generation components.<n>The model is pretrained across more than 20 embodiments, processing over one million robot trajectories per epoch.<n>It achieves non-trivial zero-shot real-robot behavior, completing several tasks, including a held-out deformable manipulation task, at high task progress on a 17-task suite.
  arXiv  Detail & Related papers  (2026-05-29T06:04:03Z)
• Primitive Subspaces Mediate Few-Shot Transfer in VLAs [1.3918848543076061]
  We investigate whether primitive-aware training produces a transferable artifact.<n>We train two VLA architectures with different inductive biases.<n>We show few-shot transfer degrades by 32 percentage points while ablating a random subspace of equal dimensionality has no effect.
  arXiv  Detail & Related papers  (2026-05-29T00:37:53Z)
• ST4VLA: Spatially Guided Training for Vision-Language-Action Models [80.35847468618276]
  Large vision-language models (VLMs) excel at multimodal understanding but fall short when extended to embodied tasks.<n>We introduce ST4VLA, a dual-system Vision-Language-Action framework to align action learning with spatial priors inVLMs.<n> Empirically, ST4VLA achieves substantial improvements over vanilla VLA, with performance increasing from 66.1 -> 84.6 on Google Robot and from 54.7 -> 73.2 on WidowX Robot.
  arXiv  Detail & Related papers  (2026-02-10T18:59:17Z)
• Rethinking Visual-Language-Action Model Scaling: Alignment, Mixture, and Regularization [65.37179698521766]
  Vision-Language-Action (VLA) models show strong promise for generalist robot control.<n>It remains unclear whether -- and under what conditions -- the standard "scale data" recipe translates to robotics.<n>We present a systematic, controlled study of VLA scaling that revisits core training choices for pretraining across diverse robots.
  arXiv  Detail & Related papers  (2026-02-10T12:25:43Z)
• EVOLVE-VLA: Test-Time Training from Environment Feedback for Vision-Language-Action Models [57.75717492488268]
  Vision-Language-Action (VLA) models have advanced robotic manipulation by leveraging large language models.<n>Supervised Finetuning (SFT) requires hundreds of demonstrations per task, rigidly memorizing trajectories, and failing to adapt when deployment conditions deviate from training.<n>We introduce EVOLVE-VLA, a test-time training framework enabling VLAs to continuously adapt through environment interaction with minimal or zero task-specific demonstrations.
  arXiv  Detail & Related papers  (2025-12-16T18:26:38Z)
• Detect Anything via Next Point Prediction [51.55967987350882]
  Rex- Omni is a 3B-scale MLLM that achieves state-of-the-art object perception performance.<n>On benchmarks like COCO and LVIS, Rex- Omni attains performance comparable to or exceeding regression-based models.
  arXiv  Detail & Related papers  (2025-10-14T17:59:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.