Related papers: FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies

URL: http://arxiv.org/abs/2605.27284v1
Date: Tue, 26 May 2026 17:01:10 GMT
Title: FineVLA: Fine-Grained Instruction Alignment for Steerable Vision-Language-Action Policies
Abstract summary: FineVLA is an open framework for action-aligned fine-grained VLA supervision.<n>It unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets.<n>It includes a benchmark with 500 videos, 10,816 atomic facts, and 1,030 VQA questions.
Score: 26.96931937675281
License: http://creativecommons.org/licenses/by-sa/4.0/
Abstract: Vision-Language-Action (VLA) models are increasingly expected to not only complete robot tasks, but also follow human instructions about how those tasks should be executed. However, existing robot datasets usually pair trajectories with coarse goal-level language, leaving execution-critical details such as active arm, approach direction, and contact region unspecified. This limits steerable policy learning and robotic video understanding. We introduce FineVLA, an open framework for action-aligned fine-grained VLA supervision. The framework includes: (1) a data construction tool that unifies 972,247 trajectories across 85K tasks from 10 open-source robot datasets and builds FineVLA-Data, a human-verified dataset of 47,159 fine-grained trajectories; (2) a held-out benchmark with 500 videos, 10,816 atomic facts, and 1,030 VQA questions; (3) a robotics-specialized VLM annotator for scalable fine-grained annotation; and (4) a steerable VLA policy trained with controlled mixtures of fine-grained and raw goal-level instructions. Our experiments yield three findings. First, fine-grained supervision does not sacrifice goal-level success: FG-only improves over Raw-only by +1.4 to +8.1 success-rate points across settings. Second, fine-grained and raw instructions are complementary, following a consistent inverted-U trend peaking at FG:Raw = 1:2 to 1:1. The best mixed setting reaches 86.8%/82.5% in RoboTwin simulation and 62.7/100 in real-world dual-arm manipulation (vs. 49.9 Raw-only). Third, fine-grained supervision improves steerable control: the largest real-world gains appear on pose (+23), color (+18), and approach direction (+18)--factors where goal-level instructions provide no guidance. Overall, fine-grained language should augment goal-level instructions: specifying how to execute alongside what to achieve. Project page: https://finevla.xlang.ai/

Related papers

GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation [63.853929983870955]
We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all from scratch on manipulation data.<n>It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM)<n>The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action.
arXiv Detail & Related papers (2026-09-04T17:16:48Z)
Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models [120.24139942108405]
Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models.<n>Under a fixed robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge.<n>We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment robot data.
arXiv Detail & Related papers (2026-08-27T17:59:40Z)
G0.5: One Autoregressive Stream for Robot Reasoning and Action [43.43377463558725]
We introduce G0.5, a pretrained autoregressive VLA in which a single transformer decoder emits reasoning and action tokens under a single objective.<n>Because reasoning and action share a single set of weights, the pretrained VLM's capabilities carry over to physical behavior.<n>G0.5 surpasses state-of-the-art models across 7 independent regimes.
arXiv Detail & Related papers (2026-08-12T07:26:47Z)
StellaVLA: In-Context Structured Demonstration for Generalizable Vision-Language-Action Models [37.45050514108603]
We present StellaVLA, a framework that adapts at test time by conditioning on a single retrieved demonstration.<n>An automated offline pipeline converts each raw trajectory into a structured demonstration, e.g., a task plan, sub-goal descriptions, and verbalized 3D motion, at zero human-annotation cost.<n>A parallel dual-training design internalizes this reasoning during training through a joint action-and-language objective, while inference uses the action expert alone.
arXiv Detail & Related papers (2026-08-12T05:30:53Z)
Uni-LaViRA: Language-Vision-Robot Actions Translation for Unified Embodied Navigation [60.07205156194741]
Embodied navigation requires an agent to map language and visual observations to a stream of spatial actions that drive a real robot through environments it has never seen.<n>We present Uni-LaViRA, a unified agentic architecture that extends the same insight to four task families and to four heterogeneous real robots.
arXiv Detail & Related papers (2026-05-26T18:52:04Z)
Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation [5.967530183571141]
Human-to-Robot'' imitation learning pipeline enables robots to acquire manipulation skills directly from unstructured video demonstrations.<n>Key innovation is a modular framework that decouples the learning process into two distinct stages.<n>For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations.
arXiv Detail & Related papers (2026-02-22T13:26:27Z)
ST4VLA: Spatially Guided Training for Vision-Language-Action Models [80.35847468618276]
Large vision-language models (VLMs) excel at multimodal understanding but fall short when extended to embodied tasks.<n>We introduce ST4VLA, a dual-system Vision-Language-Action framework to align action learning with spatial priors inVLMs.<n> Empirically, ST4VLA achieves substantial improvements over vanilla VLA, with performance increasing from 66.1 -> 84.6 on Google Robot and from 54.7 -> 73.2 on WidowX Robot.
arXiv Detail & Related papers (2026-02-10T18:59:17Z)
BridgeVLA: Input-Output Alignment for Efficient 3D Manipulation Learning with Vision-Language Models [37.699828966838986]
BridgeVLA is a novel 3D VLA model that projects 3D inputs to multiple 2D images, ensuring input alignment with the VLM backbone.<n>It utilizes 2D heatmaps for action prediction, unifying the input and output spaces within a consistent 2D image space.<n>It is able to achieve a success rate of 96.8% on 10+ tasks with only 3 trajectories per task, highlighting its extraordinary sample efficiency.
arXiv Detail & Related papers (2025-06-09T17:36:34Z)
RoboRefer: Towards Spatial Referring with Reasoning in Vision-Language Models for Robotics [67.11221574129937]
Spatial referring is a fundamental capability of embodied robots to interact with the 3D physical world.<n>We propose RoboRefer, a 3D-aware VLM that can first achieve precise spatial understanding.<n>RoboRefer advances generalized multi-step spatial reasoning via reinforcement fine-tuning.
arXiv Detail & Related papers (2025-06-04T17:59:27Z)
HAMSTER: Hierarchical Action Models For Open-World Robot Manipulation [54.03004125910057]
We show that hierarchical vision-language-action models can be more effective in utilizing off-domain data than standard monolithic VLA models.<n>We show that, with the hierarchical design, the high-level VLM can transfer across significant domain gaps between the off-domain finetuning data and real-robot testing scenarios.
arXiv Detail & Related papers (2025-02-08T07:50:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.