ACE: Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning

• URL: http://arxiv.org/abs/2607.04162v1
• Date: Sun, 05 Jul 2026 08:07:42 GMT
• Title: ACE: Agentic Control for Embodied Manipulation via Zero-shot Workflow Reasoning
• Abstract summary: We present ACE, a zero-shot workflow reasoning framework for tabletop pick-and-place from natural language.<n>We evaluate ACE on logically complex, long-horizon tasks, including zero-shot multi-step equation formation with number cubes and constraint-based object retrieval.
• Score: 32.18940485472935
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Open-ended tabletop manipulation requires agents to not only understand natural language but also adapt to dynamic environments and execution failures. We present ACE (Agentic Control for Embodied Manipulation), a zero-shot workflow reasoning framework for tabletop pick-and-place from natural language. Rather than relying on direct low-level action mapping, ACE combines agentic workflow reasoning with two robot-facing executable skills: a visual grounding interface and a reusable pick-and-place primitive. To bridge semantic reasoning and physical control, the active sub-goal is grounded into a mask-mediated vision-action interface. This unified mask specifies the target object and destination, is tracked over time, exposed for human verification, and ultimately passed to a task-agnostic downstream policy for execution. Crucially, ACE operates in a closed loop supported by a multi-timescale memory. After an action is executed, the system automatically verifies whether the intended sub-goal succeeded, using the outcome to advance, retry, repair, or replan. This enables online adaptation to user corrections, scene changes, and physical failures. We evaluate ACE on logically complex, long-horizon tasks, including zero-shot multi-step equation formation with number cubes and constraint-based object retrieval. ACE demonstrates task-level zero-shot generalization on novel semantic constraints and randomized tabletop scenes without task-specific retraining. Specifically, while standard end-to-end baselines struggle to complete these logically demanding tasks, ACE achieves a 50% success rate in equation formation and a 70% success rate in constraint retrieval. This contrast demonstrates that explicit workflow reasoning and mask-mediated control offer a robust, practical route toward adaptable robotic manipulation.

Related papers

• EmbodiedSkills: A Unified Framework for Orchestrating, Training, and Deploying VLA Agents [56.25120535601186]
  Vision-language-action (VLA) models map visual observations and language instructions directly to robot actions.<n>We propose EmbodiedSkills, a framework that treats each skill decision as an execution proposal.<n>A shared executable-skill interface connects high-level skill selection, bounded low-level VLA execution, and post-action verification.
  arXiv  Detail & Related papers  (2026-09-01T14:14:47Z)
• Bridging Semantics and Physics with Constrained LLMs for Safe and Trustworthy Robotic Manipulation [6.975054201075641]
  A language-guided robot must do more than produce a plan that appears correct.<n>Plan may appear valid linguistically while being physically infeasible under kinematic and collision constraints.<n>We bridge this gap by formalizing the reasoning-execution boundary as a typed contract.
  arXiv  Detail & Related papers  (2026-08-29T17:26:20Z)
• PhyAgentOS: A Self-Evolving Operating System for Embodied Agents with Decoupled Cognitive Planning and Physical Execution [49.776611937968]
  We present PhyAgentOS, a foundation delivering scheduling, verification, memory, benchmarking, and safety as system-level services.<n>Its Session-Centered treats a session, not an action, as the minimum unit of scheduling, compatibility, supervised execution, evidence collection, and acceptance.<n>SessionVerifier distinguishes execution termination from semantic task completion via evidence-grounded verdicts of success, failure, or replan.<n> Benchmarking reuses the deployment session and verification path, so results trace to real execution.
  arXiv  Detail & Related papers  (2026-07-18T04:46:53Z)
• IMBench: A Benchmark for Intuitive Robotic Manipulation [11.843368723732679]
  Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.<n>Existing benchmarks fail to capture this integration.<n>We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution.
  arXiv  Detail & Related papers  (2026-07-17T05:34:29Z)
• ACE-Brain-0.5: A Unified Embodied Foundational Model for Physical Agentic AI [110.18670135256235]
  Embodied AI is moving from isolated perception or action modules toward physical agents that understand, plan under goals, monitor progress, and improve from experience.<n>We present ACE-Brain-0.5, a unified embodied foundation model that organizes robot intelligence into five coupled functions.<n>A single 8B backbone instantiates the first four functions: grounding objects and affordances, reasoning over 3D and egocentric spatial relations, decomposing instructions into subgoals, generating navigation and manipulation actions, and estimating progress for verification and recovery.
  arXiv  Detail & Related papers  (2026-07-05T17:43:06Z)
• SpatialClaw: Rethinking Action Interface for Agentic Spatial Reasoning [99.49739831030388]
  We study how the design of this interface shapes the agent's capacity for open-ended spatial reasoning.<n>We propose SpatialClaw, a training-free framework for spatial reasoning that adopts code as the action interface.
  arXiv  Detail & Related papers  (2026-06-11T17:59:36Z)
• PRTS: A Primitive Reasoning and Tasking System via Contrastive Representations [66.94988600664574]
  We present textbfPRTS (textbfPrimitive textbfReasoning and textbfTasking textbfSystem), a VLA foundation model that reformulates pretraining through Goal-Conditioned Reinforcement Learning.
  arXiv  Detail & Related papers  (2026-04-30T06:14:02Z)
• RADAR: Closed-Loop Robotic Data Generation via Semantic Planning and Autonomous Causal Environment Reset [48.645870795753105]
  We introduce Robust Autonomous Data Acquisition for Robotics (RADAR)<n>RADAR is a fully autonomous, closed-loop data generation engine that completely removes human intervention from the collection cycle.<n>In simulation, our framework achieves up to 90% success rates on complex, long-horizon tasks.
  arXiv  Detail & Related papers  (2026-03-12T11:18:52Z)
• EmboAlign: Aligning Video Generation with Compositional Constraints for Zero-Shot Manipulation [17.037238350207513]
  Video generative models (VGMs) pretrained on large-scale internet data can produce temporally coherent rollout videos.<n>We present method, a data-free framework that aligns VGM outputs with insight generated by vision-language models at inference time.<n>We evaluate method on six real-robot manipulation tasks requiring precise, constraint-sensitive execution.
  arXiv  Detail & Related papers  (2026-03-05T23:31:56Z)
• Learning Object-Centric Spatial Reasoning for Sequential Manipulation in Cluttered Environments [0.0]
  We present Unveiler, a framework that separates high-level spatial reasoning from low-level action execution.<n>We demonstrate this decoupled architecture is more computationally efficient in terms of parameter count and inference time.<n>Our results, achieving up to 97.6% success in partially occluded and 90.0% in fully occluded scenarios in simulation, make a case for the power of specialized, object-centric reasoning in complex manipulation tasks.
  arXiv  Detail & Related papers  (2026-03-03T01:45:53Z)
• UniManip: General-Purpose Zero-Shot Robotic Manipulation with Agentic Operational Graph [23.060488218180936]
  We present UniManip, a framework grounded in a Bi-level Agentic Operational Graph (AOG)<n>By coupling a high-level Agentic Layer for task orchestration with a low-level Scene Layer for dynamic state representation, the system continuously aligns abstract planning with geometric constraints.<n>Experiments validate the system's robust zero-shot capability on unseen objects and tasks, demonstrating a 22.5% and 25.0% higher success rate compared to state-of-the-art VLA and hierarchical baselines.
  arXiv  Detail & Related papers  (2026-02-13T16:47:26Z)
• V-CAGE: Context-Aware Generation and Verification for Scalable Long-Horizon Embodied Tasks [6.820118518027692]
  V-CAGE is a closed-loop framework for generating semantically aligned manipulation datasets at scale.<n>We propose a context-aware instantiation mechanism that enforces geometric consistency during scene synthesis.<n>We also employ a hierarchical instruction decomposition module to bridge the gap between abstract intent and low-level control.
  arXiv  Detail & Related papers  (2026-01-21T16:41:51Z)
• Geometrically-Constrained Agent for Spatial Reasoning [53.93718394870856]
  Vision Language Models exhibit a fundamental semantic-to-geometric gap in spatial reasoning.<n>Current paradigms fail to bridge this gap.<n>We propose a training-free agentic paradigm that resolves this gap by introducing a formal task constraint.
  arXiv  Detail & Related papers  (2025-11-27T17:50:37Z)
• Visual Backdoor Attacks on MLLM Embodied Decision Making via Contrastive Trigger Learning [89.1856483797116]
  We introduce BEAT, the first framework to inject visual backdoors into MLLM-based embodied agents.<n>Unlike textual triggers, object triggers exhibit wide variation across viewpoints and lighting, making them difficult to implant reliably.<n>BEAT achieves attack success rates up to 80%, while maintaining strong benign task performance.
  arXiv  Detail & Related papers  (2025-10-31T16:50:49Z)
• CAPE: Corrective Actions from Precondition Errors using Large Language Models [8.547766794082184]
  We propose a novel approach that attempts to propose corrective actions to resolve precondition errors during planning. CAPE improves the quality of generated plans by leveraging few-shot reasoning from action preconditions. Our improvements transfer to a Boston Dynamics Spot robot with a set of skills (specified in language) and associated preconditions, where CAPE improves the correctness metric of the executed task plans by 76.49% compared to SayCan.
  arXiv  Detail & Related papers  (2022-11-17T23:14:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.