Long-Horizon Manipulation via Trace-Conditioned VLA Planning

• URL: http://arxiv.org/abs/2604.21924v1
• Date: Thu, 23 Apr 2026 17:59:04 GMT
• Title: Long-Horizon Manipulation via Trace-Conditioned VLA Planning
• Abstract summary: LoHo-Manip is a framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM.<n>LoHo-Manip predicts a progress-aware remaining plan that combines a subtask sequence with an explicit done + remaining split as lightweight language memory.<n>The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control.
• Score: 55.80061850746898
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Long-horizon manipulation remains challenging for vision-language-action (VLA) policies: real tasks are multi-step, progress-dependent, and brittle to compounding execution errors. We present LoHo-Manip, a modular framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM. The manager is decoupled from the executor and is invoked in a receding-horizon manner: given the current observation, it predicts a progress-aware remaining plan that combines (i) a subtask sequence with an explicit done + remaining split as lightweight language memory, and (ii) a visual trace -- a compact 2D keypoint trajectory prompt specifying where to go and what to approach next. The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control by following the trace. Crucially, predicting the remaining plan at each step yields an implicit closed loop: failed steps persist in subsequent outputs, and traces update accordingly, enabling automatic continuation and replanning without hand-crafted recovery logic or brittle visual-history buffers. Extensive experiments spanning embodied planning, long-horizon reasoning, trajectory prediction, and end-to-end manipulation in simulation and on a real Franka robot demonstrate strong gains in long-horizon success, robustness, and out-of-distribution generalization. Project page: https://www.liuisabella.com/LoHoManip

Related papers

• Explicit Language Memory for Long-Horizon Planning in Vision-Language-Action Models [4.644589778994777]
  Vision-language-action (VLA) models provide a unified paradigm for connecting visual perception, language understanding, and robotic control.<n>We propose a hierarchical long-horizon VLA architecture with an explicit language-memory module.<n>We evaluate the proposed method in multiple simulation environments and conduct sim-to-real experiments on a real robotic platform.
  arXiv  Detail & Related papers  (2026-08-05T12:32:15Z)
• TempAct: Advancing Temporal Plausibility in Autoregressive Video Generation via Planner-Executor RL [72.7712729541565]
  Autoregressive (AR) video diffusion models enable low-latency streaming generation by synthesizing videos chunk by chunk with cached visual context.<n>A single global prompt does not specify which sub-event should be realized in each chunk, while naively switching to step-wise prompts often leads to delayed reactions, blended step semantics, and error propagation across prompt transitions.<n>We address these challenges with TempAct, a planner--executor reinforcement learning framework that jointly optimize temporal decomposition and step-conditioned execution for temporally plausible AR video generation.
  arXiv  Detail & Related papers  (2026-06-26T12:19:28Z)
• PearlVLA: Progressive Embodied Action-Plan Refinement in Latent Space [3.4519796338615225]
  We propose PearlVLA, a framework that moves deliberation into the latent space of a vision-language model (VLM)<n>A future-guided RefineNet applies scheduled residual updates to progressively refine a coarse semantic draft into a fine-grained latent action plan.<n>PearlVLA achieves state-of-the-art performance among existing methods.
  arXiv  Detail & Related papers  (2026-06-16T13:38:03Z)
• HIPIF: Hierarchical Planning and Information Folding for Long-Horizon LLM Agent Learning [51.95194664412812]
  Large Language Models (LLMs) have demonstrated strong capabilities as autonomous agents across a wide range of tasks.<n>We propose Hierarchical Planning and Information Folding (HIPIF) for long-horizon LLM agent learning.<n>HIPIF trains the agent end-to-end to organize long-horizon execution around explicit subgoals while folding completed subgoal histories to reduce long-context interference.
  arXiv  Detail & Related papers  (2026-06-09T07:35:14Z)
• Coarse-to-Control: Action-Token Planning for Vision-Language-Action Models [101.44987067974999]
  Coarse-to-Control is a plan-execute VLA that introduces planning in the action-token space.<n>We show that action-token planning consistently improves over direct action generation.
  arXiv  Detail & Related papers  (2026-06-05T10:01:37Z)
• CodeGraphVLP: Code-as-Planner Meets Semantic-Graph State for Non-Markovian Vision-Language-Action Models [13.691036198349465]
  Vision-Language-Action (VLA) models promise generalist robot manipulation, but are typically trained and deployed as short-horizon policies.<n>We present CodeGraph, a hierarchical framework that enables reliable long-horizon manipulation.
  arXiv  Detail & Related papers  (2026-04-24T05:27:27Z)
• Goal2Skill: Long-Horizon Manipulation with Adaptive Planning and Reflection [15.30405243180468]
  Long-horizon embodied manipulation tasks require persistent memory, adaptive task decomposition, and explicit recovery from execution failures.<n>Our framework explicitly separates high-level semantic reasoning from low-level motor execution.<n>A high-level planner, implemented as a VLM-based agentic module, maintains structured task memory.<n>A low-level executor, instantiated as a VLA-based visuomotor controller, carries out each sub-task.
  arXiv  Detail & Related papers  (2026-04-15T14:53:09Z)
• ProgressVLA: Progress-Guided Diffusion Policy for Vision-Language Robotic Manipulation [57.07494675832939]
  Most existing vision-language-action (VLA) models for robotic manipulation lack progress awareness.<n>In this work, we investigate the estimation and integration of task progress, proposing a novel model named textbf vla<n>Experiments on the CALVIN and LIBERO benchmarks, alongside real-world robot deployment, consistently demonstrate substantial improvements in success rates.
  arXiv  Detail & Related papers  (2026-03-29T12:38:11Z)
• See, Plan, Rewind: Progress-Aware Vision-Language-Action Models for Robust Robotic Manipulation [59.07792608884117]
  We introduce See, Plan, Rewind (SPR), a progress-aware vision-language-action framework.<n>SPR operates through a continuous core cycle, Seeing the current state and upcoming milestone, Planning a trajectory towards the next 2D waypoint, and Rewinding to a recoverable state upon failure.<n>SPR achieves state-of-the-art robustness with the smallest performance drop, surpassing OpenVLA-OFT and UniVLA.
  arXiv  Detail & Related papers  (2026-03-10T07:22:51Z)
• Critic in the Loop: A Tri-System VLA Framework for Robust Long-Horizon Manipulation [5.339854280045898]
  Critic in the Loop is an adaptive hierarchical framework driven by dynamic VLM-Expert scheduling.<n>At its core is a bionic Tri-System architecture comprising a VLM brain for global reasoning, a VLA cerebellum for reactive execution, and a lightweight visual Critic.<n>Our architecture seamlessly integrates human-inspired rules to intuitively break infinite retry loops.
  arXiv  Detail & Related papers  (2026-03-05T13:55:33Z)
• Non-Markovian Long-Horizon Robot Manipulation via Keyframe Chaining [56.62125584296097]
  Keyframe-Chaining VLA is a framework that extracts and links key historical frames to model long-horizon dependencies.<n>We design a progress-aware mechanism that dynamically retrieves historical frames based on their temporal relevance to the current execution phase.<n>We introduce a suite of four Non-Markovian manipulation tasks built upon the ManiSkill simulator to measure task success rates.
  arXiv  Detail & Related papers  (2026-03-02T05:26:29Z)
• PALM: Progress-Aware Policy Learning via Affordance Reasoning for Long-Horizon Robotic Manipulation [27.791908160098625]
  PALM builds policy learning around interaction-centric affordance reasoning and subtask progress cues.<n>Palm consistently outperforms baselines in simulation and real-world experiments.
  arXiv  Detail & Related papers  (2026-01-11T21:00:58Z)
• Efficient Robotic Policy Learning via Latent Space Backward Planning [17.770562202624962]
  Current robotic planning methods often rely on predicting multi-frame images with full pixel details.<n>We propose a Latent Space Backward Planning scheme (LBP), which begins by grounding the task into final latent goals.<n>We show that LBP outperforms existing fine-grained and forward planning methods, achieving SOTA performance.
  arXiv  Detail & Related papers  (2025-05-11T06:13:51Z)
• Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation [12.077740860502878]
  Embodied long-horizon manipulation requires robotic systems to process multimodal inputs-such as vision and natural language-and translate them into executable actions.<n>Recent methods have explored using large language models (LLMs) as high-level planners that decompose tasks into subtasks using natural language and guide pretrained low-level controllers.<n>Our framework achieves state-of-the-art performance on 30+ diverse seen and unseen long-horizon tasks across LoHoRavens, CALVIN, Franka Kitchen, and cluttered real-world settings.
  arXiv  Detail & Related papers  (2025-03-27T20:32:58Z)
• PlanDQ: Hierarchical Plan Orchestration via D-Conductor and Q-Performer [47.924941959320996]
  We propose a hierarchical planner designed for offline RL called PlanDQ. PlanDQ incorporates a diffusion-based planner at the high level, named D-Conductor, which guides the low-level policy through sub-goals. At the low level, we used a Q-learning based approach called the Q-Performer to accomplish these sub-goals.
  arXiv  Detail & Related papers  (2024-06-10T20:59:53Z)
• Modular Deep Reinforcement Learning for Continuous Motion Planning with Temporal Logic [59.94347858883343]
  This paper investigates the motion planning of autonomous dynamical systems modeled by Markov decision processes (MDP) The novelty is to design an embedded product MDP (EP-MDP) between the LDGBA and the MDP. The proposed LDGBA-based reward shaping and discounting schemes for the model-free reinforcement learning (RL) only depend on the EP-MDP states.
  arXiv  Detail & Related papers  (2021-02-24T01:11:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.