WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation

• URL: http://arxiv.org/abs/2606.26443v1
• Date: Wed, 24 Jun 2026 23:13:56 GMT
• Title: WatchAct: A Benchmark for Behavior-Grounded Robot Manipulation
• Abstract summary: WatchAct is a benchmark for robot manipulation grounded in observed human behavior.<n>WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains.<n>In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct.
• Score: 49.31386176285509
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: A robot working alongside people must reason about what they have done, in what order, and with what intent. Video carries the spatial layouts, object histories, and gestures that language leaves underspecified, yet today's manipulation benchmarks pair an instruction with a single current image, offering no way to evaluate reasoning over observed human behavior. We introduce WatchAct, a benchmark for robot manipulation grounded in observed human behavior. Each instance pairs a real-world human-action video and a language instruction with an aligned simulator scene and an executable LIBERO task, enabling scalable and reproducible evaluation. WatchAct comprises 3,000 long-horizon instances across 14 tasks in four capability domains drawn from the cognitive demands of watching another agent: parsing events (Event Grounding), recovering procedural structure (Procedural Reasoning), inferring unstated intent (Implicit Intent Inference), and tracking how the scene was changed (Episodic Reasoning). We further propose a disentangled evaluation protocol that separately measures (i)~video-to-plan reasoning by vision-language models, (ii)~policy execution under oracle plans, and (iii)~full task completion by integrated planner--policy pipelines. In both simulation and on a Franka Research 3 robot, current systems remain far from solving WatchAct. The best pipeline, Gemini-3.1-Pro with $π_{0.5}$, reaches only 16.3% Success Rate (SR) in simulation and 14.0% on the real robot. Gemini-3.1-Pro attains just 36.8% Plan SR (vs. 97.1% for humans), while $π_{0.5}$ reaches only 21.5% Task SR under oracle plans and drops to 10.6% on out-of-domain scenarios. Dataset and code are available at https://baiqi-li.github.io/watchact_page/.

Related papers

• A Brain-inspired Hierarchical Framework for Zero-Shot Robot Task Reasoning and Execution [9.745144598922584]
  Method is a zero-shot hierarchical framework inspired by the division of roles in the human brain.<n>It grounds commands in explicit object states, composes reusable atomic actions into task-conditioned sequences, and ranks alternative sequences by execution cost.<n>Method achieves higher success rates than ReKep, Dream2Flow, and $_0.5$ benchmarks.
  arXiv  Detail & Related papers  (2026-09-05T08:59:04Z)
• GE-Act 2.0: Pretraining and Scaling a World-Action Model for Robotic Manipulation [63.853929983870955]
  We introduce Genie Envisioner Act 2.0 (GE-Act 2.0), a world-action model whose trainable generative and action components are all from scratch on manipulation data.<n>It combines a control-oriented autoencoder (CoAE), a single-step visual planner (SVP), and an inverse dynamics model (IDM)<n>The components are then jointly trained with knowledge-aligned selective optimization (KASO), which reduces mismatched supervision by selecting only predicted futures judged behaviorally compatible with the recorded action.
  arXiv  Detail & Related papers  (2026-09-04T17:16:48Z)
• Zero-WAM: In-Context World-Action Modeling from Human Videos for Open-Ended Task Generalization [66.06331553787281]
  We present Zero-WAM, a causal video-action model that executes unseen tasks by following in-context human video guidance.<n>We also present HumanGen, a dataset of 74.2K human-robot ICL pairs across 8.6K tasks.<n>On seven unseen tasks in RoboTwin 2.0 simulation, Zero-WAM achieves a 47.0% average success rate, an absolute improvement of 29.5 percentage points over the strongest video-action baseline.
  arXiv  Detail & Related papers  (2026-08-26T17:59:34Z)
• The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction [23.90153863744227]
  We introduce The Imitator Game, a four-level benchmark for learning from human video.<n>It progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice.<n>We pair it with IG-10K, the largest environment-aligned paired human-robot dataset to date, and Imitator Arena, an open platform for blind A/B human evaluation.
  arXiv  Detail & Related papers  (2026-08-23T08:51:01Z)
• SABER: A Scalable Action-Based Embodied Dataset for Real-World VLA Adaptation [0.0]
  SABER is a high-fidelity retail robotics action dataset built from over 100 hours of natural in-store capture.<n>It contains 44.8K training samples across three action representation streams.<n>It yields a mean success rate of 29.3% across ten retail manipulation tasks.
  arXiv  Detail & Related papers  (2026-05-10T15:51:01Z)
• Instruct2Act: From Human Instruction to Actions Sequencing and Execution via Robot Action Network for Robotic Manipulation [14.833622989644352]
  We develop a lightweight, fully on-device pipeline that converts natural-language commands into reliable manipulation.<n>Instruct2Act attains 91.5% sub-actions prediction accuracy while retaining a small footprint.<n>Results demonstrate that fine-grained instruction-to-action parsing, coupled with DATRN-based trajectory generation and vision-guided grounding, provides a practical path to deterministic, real-time manipulation.
  arXiv  Detail & Related papers  (2026-02-10T16:25:39Z)
• RoboPaint: From Human Demonstration to Any Robot and Any View [9.083647729839688]
  We propose a Real-Sim-Real data collection and data editing pipeline to transform human demonstrations into robot-executable, environment-specific training data.<n>We show that retargeted dex-hand trajectories achieve an 84% success rate across 10 diverse object manipulation tasks.<n>We offer a scalable, cost-effective alternative to teleoperation with minimal performance loss for complex dexterous manipulation.
  arXiv  Detail & Related papers  (2026-02-05T05:45:12Z)
• Local Policies Enable Zero-shot Long-horizon Manipulation [80.1161776000682]
  We introduce ManipGen, which leverages a new class of policies for sim2real transfer: local policies.<n>ManipGen outperforms SOTA approaches such as SayCan, OpenVLA, LLMTrajGen and VoxPoser across 50 real-world manipulation tasks by 36%, 76%, 62% and 60% respectively.
  arXiv  Detail & Related papers  (2024-10-29T17:59:55Z)
• Track2Act: Predicting Point Tracks from Internet Videos enables Generalizable Robot Manipulation [65.46610405509338]
  We seek to learn a generalizable goal-conditioned policy that enables zero-shot robot manipulation. Our framework,Track2Act predicts tracks of how points in an image should move in future time-steps based on a goal. We show that this approach of combining scalably learned track prediction with a residual policy enables diverse generalizable robot manipulation.
  arXiv  Detail & Related papers  (2024-05-02T17:56:55Z)
• Self-Improving Robots: End-to-End Autonomous Visuomotor Reinforcement Learning [54.636562516974884]
  In imitation and reinforcement learning, the cost of human supervision limits the amount of data that robots can be trained on. In this work, we propose MEDAL++, a novel design for self-improving robotic systems. The robot autonomously practices the task by learning to both do and undo the task, simultaneously inferring the reward function from the demonstrations.
  arXiv  Detail & Related papers  (2023-03-02T18:51:38Z)
• Reasoning with Scene Graphs for Robot Planning under Partial Observability [7.121002367542985]
  We develop an algorithm called scene analysis for robot planning (SARP) that enables robots to reason with visual contextual information. Experiments have been conducted using multiple 3D environments in simulation, and a dataset collected by a real robot.
  arXiv  Detail & Related papers  (2022-02-21T18:45:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.