Agent as Policy for Robotic Manipulation

• URL: http://arxiv.org/abs/2609.12541v2
• Date: Mon, 14 Sep 2026 08:11:02 GMT
• Title: Agent as Policy for Robotic Manipulation
• Abstract summary: We introduce Agent as Policy, which places task planning and execution under the agent's control.<n>We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects.<n>AGP succeeds in at least eight of ten trials for each evaluated task configuration.
• Score: 27.777226380588928
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We demonstrate that a general-purpose agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. Across assembly, block construction, and dice flipping, AGP succeeds in at least eight of ten trials for each evaluated task configuration. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.

Related papers

• IMBench: A Benchmark for Intuitive Robotic Manipulation [11.843368723732679]
  Humans combine reasoning and motor control to solve complex manipulation tasks under diverse constraints.<n>Existing benchmarks fail to capture this integration.<n>We introduce IMBENCH, a benchmark designed to evaluate intuitive manipulation as an integrated capability spanning perception, physical reasoning, action generation, and iterative execution.
  arXiv  Detail & Related papers  (2026-07-17T05:34:29Z)
• Behavior Prompting Policy: Demonstrations as Prompts for Manipulation [12.69446064622751]
  We study behavior prompting, a paradigm that enables robots to perform new tasks at inference time given a single human demonstration.<n>We present contributions in algorithm, data, and evaluation.
  arXiv  Detail & Related papers  (2026-06-29T15:23:44Z)
• Acting and Planning with Hierarchical Operational Models on a Mobile Robot: A Study with RAE+UPOM [5.758011837296545]
  We present the first physical deployment of an integrated actor-planner system that shares hierarchical operational models for both acting and planning.<n>We implement RAE+UPOM on a mobile manipulator in a real-world deployment for an object collection task.
  arXiv  Detail & Related papers  (2025-07-15T14:20:26Z)
• COHERENT: Collaboration of Heterogeneous Multi-Robot System with Large Language Models [49.24666980374751]
  COHERENT is a novel LLM-based task planning framework for collaboration of heterogeneous multi-robot systems.<n>A Proposal-Execution-Feedback-Adjustment mechanism is designed to decompose and assign actions for individual robots.<n>The experimental results show that our work surpasses the previous methods by a large margin in terms of success rate and execution efficiency.
  arXiv  Detail & Related papers  (2024-09-23T15:53:41Z)
• Make-An-Agent: A Generalizable Policy Network Generator with Behavior-Prompted Diffusion [41.52811286996212]
  Make-An-Agent is a novel policy parameter generator for behavior-to-policy generation.<n>We show how it can generate a control policy for an agent using just one demonstration of desired behaviors as a prompt.<n>We also deploy policies generated by Make-An-Agent onto real-world robots on locomotion tasks.
  arXiv  Detail & Related papers  (2024-07-15T17:59:57Z)
• Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation [65.46610405509338]
  We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal. We show that this approach of combining scalably learned track prediction with a residual policy enables diverse generalizable robot manipulation.
  arXiv  Detail & Related papers  (2024-05-02T17:56:55Z)
• Learning to Act from Actionless Videos through Dense Correspondences [87.1243107115642]
  We present an approach to construct a video-based robot policy capable of reliably executing diverse tasks across different robots and environments. Our method leverages images as a task-agnostic representation, encoding both the state and action information, and text as a general representation for specifying robot goals. We demonstrate the efficacy of our approach in learning policies on table-top manipulation and navigation tasks.
  arXiv  Detail & Related papers  (2023-10-12T17:59:23Z)
• RoboAgent: Generalization and Efficiency in Robot Manipulation via Semantic Augmentations and Action Chunking [54.776890150458385]
  We develop an efficient system for training universal agents capable of multi-task manipulation skills. We are able to train a single agent capable of 12 unique skills, and demonstrate its generalization over 38 tasks. On average, RoboAgent outperforms prior methods by over 40% in unseen situations.
  arXiv  Detail & Related papers  (2023-09-05T03:14:39Z)
• Zero-Shot Robot Manipulation from Passive Human Videos [59.193076151832145]
  We develop a framework for extracting agent-agnostic action representations from human videos. Our framework is based on predicting plausible human hand trajectories. We deploy the trained model zero-shot for physical robot manipulation tasks.
  arXiv  Detail & Related papers  (2023-02-03T21:39:52Z)
• ProgPrompt: Generating Situated Robot Task Plans using Large Language Models [68.57918965060787]
  Large language models (LLMs) can be used to score potential next actions during task planning. We present a programmatic LLM prompt structure that enables plan generation functional across situated environments.
  arXiv  Detail & Related papers  (2022-09-22T20:29:49Z)
• Synthesis and Execution of Communicative Robotic Movements with Generative Adversarial Networks [59.098560311521034]
  We focus on how to transfer on two different robotic platforms the same kinematics modulation that humans adopt when manipulating delicate objects. We choose to modulate the velocity profile adopted by the robots' end-effector, inspired by what humans do when transporting objects with different characteristics. We exploit a novel Generative Adversarial Network architecture, trained with human kinematics examples, to generalize over them and generate new and meaningful velocity profiles.
  arXiv  Detail & Related papers  (2022-03-29T15:03:05Z)
• Manipulation of Articulated Objects using Dual-arm Robots via Answer Set Programming [10.316694915810947]
  The manipulation of articulated objects is of primary importance in Robotics, and can be considered as one of the most complex manipulation tasks. Traditionally, this problem has been tackled by developing ad-hoc approaches, which lack flexibility and portability. We present a framework based on Answer Set Programming (ASP) for the automated manipulation of articulated objects in a robot control architecture.
  arXiv  Detail & Related papers  (2020-10-02T18:50:39Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.