StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models

• URL: http://arxiv.org/abs/2608.11671v1
• Date: Wed, 12 Aug 2026 05:30:53 GMT
• Title: StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models
• Abstract summary: We present StellaVLA, a framework that adapts at test time by conditioning on a single retrieved demonstration.<n>An automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost.<n>A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone.
• Score: 37.45050514108603
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Vision-Language-Action (VLA) models can follow instructions and manipulate objects, but their performance often collapses out of distribution (OOD), when the scene, viewpoint, or object differs from training. Adapting to each new situation typically requires collecting more data and fine-tuning. We present StellaVLA, a framework that instead adapts at test time by conditioning on a single retrieved demonstration. The key idea is to move beyond imitating what an expert did and instead convey why: an automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost. Provided as in-context guidance, this structured demonstration lets the policy reason about the task rather than mimic a pixel trajectory, which also makes it transferable across embodiments (real-robot, human-hand, or XR demonstrations). A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone, preserving real-time, high-frequency control with no added latency. On the VLA-Arena leaderboard(Aug 1, 2026), StellaVLA ranks first with an overall score of 0.63, versus 0.44 and 0.22 for the strong prior models ($π_{0.5}$ and LingBot-VLA), and it further leads on LIBERO with 98.8% average success rate and LIBERO-Plus with 85.1% success rate. Our real-robot benchmark demonstrates that StellaVLA can use both human/robot demos and human-to-robot (XR) demos as in-context structured demonstration to help VLA model adapt to OOD tasks.

Related papers

• Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models [120.24139942108405]
  Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models.<n>Under a fixed robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge.<n>We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment robot data.
  arXiv  Detail & Related papers  (2026-08-27T17:59:40Z)
• Learning to Move Before Learning to Do: Task-Agnostic pretraining for VLAs [46.61554174146852]
  Task-Agnostic Pretraining (TAP) is a two-stage framework that first learns transferable motor priors from cheap, unlabeled interaction data.<n>On the SIMPLER benchmark, TAP matches models trained on over 1M expert trajectories while using orders of magnitude less labeled data.<n>On a real-world WidowX platform, TAP retains 25% success under camera perturbations where internet-scale baselines collapse to 0%.
  arXiv  Detail & Related papers  (2026-07-02T17:33:37Z)
• GEAR-VLA: Learning Geometry-Aware Action Representations for Generalizable Robotic Manipulation [49.16739604572808]
  Vision-Language-Action (VLA) models achieve strong benchmark performance but struggle in real-world deployment with unseen objects.<n>We argue that this stems from the lack of a unified geometry-aware manipulation representation.<n>We propose GEAR-VLA, a VLA framework for learning unified geometry-aware action representations for generalizable robotic manipulation.
  arXiv  Detail & Related papers  (2026-06-07T09:23:16Z)
• Two Bridges, One Pathway: From VLMs to Generalizable VLAs with Embodied Trajectory-Coupled Data [93.7685703383343]
  Vision-language models (VLMs) are powerful general-purpose reasoners, yet converting them into robot control policies is surprisingly difficult.<n>We argue that this gap can be bridged gradually with the right intermediate data.<n>We show that mixing task-relevant out-of-distribution ETC data with a small amount of action data enables the model to generalize to novel visual-language conditions.
  arXiv  Detail & Related papers  (2026-06-07T08:57:51Z)
• RotVLA: Rotational Latent Action for Vision-Language-Action Model [54.22746299071677]
  We introduce RotVLA, a VLA framework built on a continuous rotational latent action representation.<n>Latent actions are modeled as elements of SO(n), providing continuity, compositionality, and structured geometry aligned with real-world action dynamics.<n>RotVLA consists of a VLM backbone and a flow-matching action head, pretrained on large-scale cross-embodiment robotic datasets and human videos.
  arXiv  Detail & Related papers  (2026-05-13T11:58:02Z)
• Universal Pose Pretraining for Generalizable Vision-Language-Action Policies [83.39008378156647]
  Existing Vision-Language-Action (VLA) models often suffer from feature collapse and low training efficiency.<n>We propose Pose-VLA, a decoupled paradigm that separates VLA training into a pre-training phase for extracting universal 3D spatial priors.<n>Our framework follows a two-stage pre-training pipeline, establishing fundamental spatial grounding via poses followed by motion alignment.
  arXiv  Detail & Related papers  (2026-02-23T11:00:08Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.