Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation

• URL: http://arxiv.org/abs/2607.05377v1
• Date: Mon, 06 Jul 2026 17:55:05 GMT
• Title: Cortex: A Bidirectionally Aligned Embodied Agent Framework for Long-horizon Manipulation
• Abstract summary: Vision-Language-Action (VLA) models struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations.<n>We introduce Cortex, a framework that conveys executable and tractable subtask plans from high-level VLM to low-level VLA.<n>This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data.
• Score: 40.0334846364089
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: While recent Vision-Language-Action (VLA) models show promise toward generalist manipulation policies, they struggle with long-horizon tasks due to their Markovian nature-relying solely on current observations. Hierarchical dual-system methods address this but suffer from a gap between high-level planning semantics and low-level execution kinematics. We introduce Cortex, a bidirectionally aligned embodied agent framework with a customized planning interface that conveys executable and tractable subtask plans from high-level VLM to low-level VLA. Specifically, we standardize manipulation subtasks into 32 canonical skill primitives and inject tractability principles, such as representative object attributes and improved trajectory reachability, into the data generation pipeline. This enables automatic annotation of over 4k hours of open-source video data and generation of 30 hours of simulation data. We further devise an event-balanced sampling strategy to construct training data for fine-tuning the framework to better handle planning ambiguity during subtask transitions, enhanced by carefully designed harness engineering from task contexts to skill constraints during inference. Both open-loop VLM and closed-loop system evaluations demonstrate Cortex's efficacy, e.g., it outperforms monolithic baselines by 3.1% on Libero-long and 4.1% on RoboTwin. Notably, Cortex's generalist VLM enables zero-shot completion of unseen real-world long-horizon tasks, such as multi-stage chemistry experiments, by simply combining with a fine-tuned VLA-a capability infeasible through VLA fine-tuning alone.

Related papers

• EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents [56.25120535601186]
  Vision-language-action (VLA) models map visual observations and language instructions directly to robot actions.<n>We propose EmbodiedSkills, a framework that treats each skill decision as an execution proposal.<n>A shared executable-skill interface connects high-level skill selection, bounded low-level VLA execution, and post-action verification.
  arXiv  Detail & Related papers  (2026-09-01T14:14:47Z)
• Beyond Data Scaling: Representation-Centric Continued Pre-training for Vision-Language-Action Models [120.24139942108405]
  Scaling robot data is crucial for building generalist Vision-Language-Action (VLA) models.<n>Under a fixed robot-data budget, continued pre-training must turn limited trajectories into transferable visual-action knowledge.<n>We propose VLAct, a VLA-oriented VLM backbone trained on broad, heterogeneous, multi-embodiment robot data.
  arXiv  Detail & Related papers  (2026-08-27T17:59:40Z)
• Triplet2Track: A Hierarchical System with Object-Centric Representations for Reliable Long-Horizon Manipulation [14.246688157481143]
  Triplet-to-Track System (TTS) is a closed-loop long-horizon imitation learning system that uses human videos to reduce reliance on robot-collected data.<n>TTS represents high-level subgoals as instance-grounded triplets, translates them into continuous track priors for execution, and monitors task progress from observations for online replanning.
  arXiv  Detail & Related papers  (2026-08-24T04:46:25Z)
• Long-Horizon Manipulation via Trace-Conditioned VLA Planning [55.80061850746898]
  LoHo-Manip is a framework that scales short-horizon VLA execution to long-horizon instruction following via a dedicated task-management VLM.<n>LoHo-Manip predicts a progress-aware remaining plan that combines a subtask sequence with an explicit done + remaining split as lightweight language memory.<n>The executor VLA is adapted to condition on the rendered trace, thereby turning long-horizon decision-making into repeated local control.
  arXiv  Detail & Related papers  (2026-04-23T17:59:04Z)
• AtomVLA: Scalable Post-Training for Robotic Manipulation via Predictive Latent World Models [9.608633915316252]
  Vision-Language-Action (VLA) models demonstrate remarkable potential for generalizable robotic manipulation.<n>Current paradigms rely on coarse, high-level task instructions during supervised fine-tuning.<n>We propose method, the first subtask-aware VLA framework integrated with a scalable offline post-training pipeline.
  arXiv  Detail & Related papers  (2026-03-09T15:52:48Z)
• Global Prior Meets Local Consistency: Dual-Memory Augmented Vision-Language-Action Model for Efficient Robotic Manipulation [95.89924101984566]
  We introduce OptimusVLA, a dual-memory VLA framework with Global Prior Memory (GPM) and Local Consistency Memory (LCM)<n>GPM replaces Gaussian noise with task-level priors retrieved from semantically similar trajectories.<n>LCM injects a learned consistency constraint that enforces temporal coherence and smoothness of trajectory.
  arXiv  Detail & Related papers  (2026-02-22T15:39:34Z)
• ColaVLA: Leveraging Cognitive Latent Reasoning for Hierarchical Parallel Trajectory Planning in Autonomous Driving [44.008287454538596]
  Vision-language models (VLMs) enrich this paradigm by introducing cross-modal priors and commonsense reasoning.<n>Current VLM-based planners face three key challenges: (i) a mismatch between discrete text reasoning and continuous control, (ii) high latency from autoregressive chain-of-thought decoding, and (iii) inefficient or non-causal planners that limit real-time deployment.<n>We propose ColaVLA, a unified vision-language-action framework that transfers reasoning from text to a unified latent space and couples it with a hierarchical, parallel trajectory decoder.
  arXiv  Detail & Related papers  (2025-12-28T14:06:37Z)
• VLA-Arena: An Open-Source Framework for Benchmarking Vision-Language-Action Models [33.143187576673775]
  We propose a novel structured task design framework to quantify difficulty across three axes.<n>For Task Structure, VLA-Arena's 170 tasks are grouped into four dimensions: Safety, Distractor, Extrapolation, and Long Horizon.<n>We provide an end-to-end toolchain from task definition to automated evaluation and the VLA-Arena-S/M/L datasets for fine-tuning.
  arXiv  Detail & Related papers  (2025-12-27T09:40:54Z)
• CronusVLA: Towards Efficient and Robust Manipulation via Multi-Frame Vision-Language-Action Modeling [84.51372201195132]
  CronusVLA is a unified framework that extends single-frame VLA models to the multi-frame paradigm.<n>CronusVLA achieves leading performance and superior robustness, with a 70.9% success rate.<n>These results highlight the potential of efficient multi-frame adaptation in VLA models for more powerful and robust real-world deployment.
  arXiv  Detail & Related papers  (2025-06-24T17:30:27Z)
• RoboCerebra: A Large-scale Benchmark for Long-horizon Robotic Manipulation Evaluation [59.9896841079005]
  We introduce RoboCerebra, a benchmark for evaluating high-level reasoning in long-horizon robotic manipulation.<n>The dataset is constructed via a top-down pipeline, where GPT generates task instructions and decomposes them into subtask sequences.<n>Compared to prior benchmarks, RoboCerebra features significantly longer action sequences and denser annotations.
  arXiv  Detail & Related papers  (2025-06-07T06:15:49Z)
• Embodied Long Horizon Manipulation with Closed-loop Code Generation and Incremental Few-shot Adaptation [12.077740860502878]
  Embodied long-horizon manipulation requires robotic systems to process multimodal inputs-such as vision and natural language-and translate them into executable actions.<n>Recent methods have explored using large language models (LLMs) as high-level planners that decompose tasks into subtasks using natural language and guide pretrained low-level controllers.<n>Our framework achieves state-of-the-art performance on 30+ diverse seen and unseen long-horizon tasks across LoHoRavens, CALVIN, Franka Kitchen, and cluttered real-world settings.
  arXiv  Detail & Related papers  (2025-03-27T20:32:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.