VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation

• URL: http://arxiv.org/abs/2606.07723v1
• Date: Fri, 05 Jun 2026 16:21:27 GMT
• Title: VoLo: A Physical Orchestrator for Open-Vocabulary Long-Horizon Manipulation
• Abstract summary: Open-vocabulary long-horizon manipulation requires robots to reason over flexible instructions and complex multi-object scenes.<n>We address these demands with a closed agent loop in which a VLM orchestrates heterogeneous robot capabilities as interruptible tools.<n>We propose VoLoAgent, a VLM that plans, monitors, and recovers by treating a VLA/WAM as an interruptible tool it steers mid-rollout.
• Score: 32.23131310342716
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Open-vocabulary long-horizon manipulation requires robots to reason over flexible instructions and complex multi-object scenes while adaptively planning, executing, monitoring, and recovering from failures. We address these demands with a closed agent loop in which a VLM orchestrates heterogeneous robot capabilities as interruptible tools. Unlike in virtual AI agents, the timing of decisions, actions and tool calls is important in a physical world that does not pause for reasoning. We refer to this setting as Physical Orchestration, and propose VoLoAgent, a VLM that plans, monitors, and recovers by treating a VLA/WAM as an interruptible tool it steers mid-rollout alongside vision models and action primitives. To evaluate these long-horizon capabilities, we introduce RoboVoLo, a high-fidelity benchmark for open-vocabulary long-horizon manipulation across common sense, memory/state tracking, complex references, and world knowledge, with both task-level success and failure-mode diagnostics. Experiments show VoLoAgent substantially outperforms single VLA/VLM or tool-based systems, with validation on real-robot experiments. Project page: https://chicychen.github.io/VoLo/

Related papers

• Show-Harness: Just a VLM Agent Can Play Robots [53.80887083886227]
  Embodied vision-language models (VLMs) exhibit broad intelligence about the world, yet translating this intelligence into robot control remains challenging.<n>We present Show-Harness, an Embodied Harness that enablesVLMs to "play" robots through a compact semantic interface linking intent to action.<n>We further develop GUMI, which extends the same semantic action space to GUI-based demonstration collection, allowing humans and agents to "play" robots across embodiments without specialized teleoperation hardware.
  arXiv  Detail & Related papers  (2026-09-09T17:53:38Z)
• ETA: A New Agentic Paradigm for Embodied Tasks [76.25893120982523]
  We introduce the Embodied Task Agent (ETA), a new paradigm for extending digital agents into the physical world.<n>ETA centers the robot around a Planner that chooses one Tool call at a time, an Interface that controls execution, and a World that returns the result and a fresh observation.<n>ETA provides replaceable Planners, composable Tools and Skills, auditable memory, replayable trajectories, and common interfaces for simulation and real robots.
  arXiv  Detail & Related papers  (2026-08-04T16:56:09Z)
• VIA: Visual Interface Agent for Robot Control [34.69686658431114]
  VIA (Visual Interface Agent for robot control) is a framework that recasts robot control as an agentic task.<n>It drives a manipulator through a browser-based 3D interface by taking screenshots, issuing intuitive commands, observing the outcome, and adjusting.<n>It achieves 96.7% success on three LIBERO-Goal tasks and 100% on a long-horizon rainbow assembly task.
  arXiv  Detail & Related papers  (2026-07-13T05:52:12Z)
• Sentinel-VLA: A Metacognitive VLA Model with Active Status Monitoring for Dynamic Reasoning and Error Recovery [62.75419724651416]
  We introduce textbfSentinel-VLA, a metacognitive VLA model equipped with an active sentinel'' module to monitor real-time execution status.<n>All training data is automatically generated and annotated through our designed pipeline.<n>Real-world experiments demonstrate that Sentinel-VLA boosts the task success rate by over 30% compared to the SOTA model, PI0.
  arXiv  Detail & Related papers  (2026-05-02T02:10:54Z)
• Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation [5.339854280045898]
  Critic in the Loop is an adaptive hierarchical framework driven by dynamic VLM-Expert scheduling.<n>At its core is a bionic Tri-System architecture comprising a VLM brain for global reasoning, a VLA cerebellum for reactive execution, and a lightweight visual Critic.<n>Our architecture seamlessly integrates human-inspired rules to intuitively break infinite retry loops.
  arXiv  Detail & Related papers  (2026-03-05T13:55:33Z)
• NovaPlan: Zero-Shot Long-Horizon Manipulation via Closed-Loop Video Language Planning [36.20611975009607]
  We introduce NovaPlan, a hierarchical framework that unifies closed-loop VLM and video planning.<n>At the high level, a VLM planner decomposes tasks into sub-goals and monitors robot execution in a closed loop.<n>To compute low-level robot actions, we extract and utilize both task-relevant object keypoints and human hand poses.
  arXiv  Detail & Related papers  (2026-02-23T18:35:18Z)
• AgenticLab: A Real-World Robot Agent Platform that Can See, Think, and Act [27.922630781100864]
  We present AgenticLab, a model-agnostic robot agent platform and benchmark for open-world manipulation.<n>We benchmark state-of-the-art VLM-based agents on real-robot tasks in unstructured environments.<n>Our benchmark reveals several failure modes that offline vision-language tests fail to capture.
  arXiv  Detail & Related papers  (2026-02-02T05:30:14Z)
• LoLA: Long Horizon Latent Action Learning for General Robot Manipulation [38.209790870296835]
  Long Horizon Latent Action Learning (LoLA) is a framework designed for robot manipulation.<n>We first employ Vision-Language Models to encode rich contextual features from historical sequences and multi-view observations.<n>We then introduce State-Aware Latent Re-representation, which transforms visual inputs and language commands into actionable robot motion space.
  arXiv  Detail & Related papers  (2025-12-23T08:45:24Z)
• DeepAgent: A General Reasoning Agent with Scalable Toolsets [111.6384541877723]
  DeepAgent is an end-to-end deep reasoning agent that performs autonomous thinking, tool discovery, and action execution.<n>To address the challenges of long-horizon interactions, we introduce an autonomous memory folding mechanism that compresses past interactions into structured episodic, working, and tool memories.<n>We develop an end-to-end reinforcement learning strategy, namely ToolPO, that leverages LLM-simulated APIs and applies tool-call advantage attribution to assign fine-grained credit to the tool invocation tokens.
  arXiv  Detail & Related papers  (2025-10-24T16:24:01Z)
• Reflective Planning: Vision-Language Models for Multi-Stage Long-Horizon Robotic Manipulation [90.00687889213991]
  Solving complex long-horizon robotic manipulation problems requires sophisticated high-level planning capabilities.<n>Vision-language models (VLMs) pretrained on Internet data could in principle offer a framework for tackling such problems.<n>In this paper, we introduce a novel test-time framework that enhancesVLMs' physical reasoning capabilities for multi-stage manipulation tasks.
  arXiv  Detail & Related papers  (2025-02-23T20:42:15Z)
• Wonderful Team: Zero-Shot Physical Task Planning with Visual LLMs [0.0]
  Wonderful Team is a framework for executing high-level robotic planning in a zero-shot regime.<n>We show that Wonderful Team's performance on real-world semantic and physical planning tasks often exceeds methods that rely on separate vision systems.
  arXiv  Detail & Related papers  (2024-07-26T21:18:57Z)
• LLaRA: Supercharging Robot Learning Data for Vision-Language Policy [56.505551117094534]
  We introduce LLaRA: Large Language and Robotics Assistant, a framework that formulates robot action policy as visuo-textual conversations.<n>First, we present an automated pipeline to generate conversation-style instruction tuning data for robots from existing behavior cloning datasets.<n>We show that a VLM finetuned with a limited amount of such datasets can produce meaningful action decisions for robotic control.
  arXiv  Detail & Related papers  (2024-06-28T17:59:12Z)
• Plan-Seq-Learn: Language Model Guided RL for Solving Long Horizon Robotics Tasks [50.27313829438866]
  Plan-Seq-Learn (PSL) is a modular approach that uses motion planning to bridge the gap between abstract language and learned low-level control. PSL achieves success rates of over 85%, out-performing language-based, classical, and end-to-end approaches.
  arXiv  Detail & Related papers  (2024-05-02T17:59:31Z)
• PIVOT: Iterative Visual Prompting Elicits Actionable Knowledge for VLMs [140.14239499047977]
  Vision language models (VLMs) have shown impressive capabilities across a variety of tasks, from logical reasoning to visual understanding. We propose a novel visual prompting approach for VLMs that we call Prompting with Iterative Visual Optimization (PIVOT) We find, perhaps surprisingly, that our approach enables zero-shot control of robotic systems without any robot training data, navigation in a variety of environments, and other capabilities.
  arXiv  Detail & Related papers  (2024-02-12T18:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.