The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction

• URL: http://arxiv.org/abs/2608.22301v1
• Date: Sun, 23 Aug 2026 08:51:01 GMT
• Title: The Imitator Game: Benchmarking Robot Imitative Ability Beyond Action Prediction
• Abstract summary: We introduce The Imitator Game, a four-level benchmark for learning from human video.<n>It progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice.<n>We pair it with IG-10K, the largest environment-aligned paired human-robot dataset to date, and Imitator Arena, an open platform for blind A/B human evaluation.
• Score: 23.90153863744227
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Humans imitate at the level of intent: given a demonstration, we infer its goal and carry it out with whatever tools, objects, and layouts are at hand. Current robot policies instead learn observation-to-action mappings from visual inputs and language instructions, without explicitly inferring the demonstrated task. Learning from human video thus remains largely trajectory-level: models can replay motions in near-identical scenes, but still struggle to imitate what the demonstrator intends rather than merely what they do. We introduce The Imitator Game, a four-level benchmark (L0-L3) that progressively widens the gap between the human demonstration and the robot's own scene, isolating where trajectory replay ceases to suffice and task understanding becomes necessary. We pair it with IG-10K, the largest environment-aligned paired human-robot dataset to date and the only one instantiated across all four levels in both real and simulated settings (20,000+ paired episodes, 50+ tasks, 6 domains), and Imitator Arena, an open platform for blind A/B human evaluation. Across nine state-of-the-art models, performance is stable from L0 to L2 but collapses at L3, identifying functional substitution - achieving the same intent through a different object affordance - as the decisive barrier to intent-level imitation. Human-video-conditioned models outperform caption-conditioned ones, yet every model falls below 13% zero-shot success on unseen tasks; fine-tuning IG-10K-pretrained models with only $10$ paired human-robot demonstrations yields large gains that grow with pretraining scale. The project website and access to Imitator Arena are available at https://imitator-game.github.io.

Related papers

• ZimaBlue: Evolving Generalizable World Action Models through Scalable Video Pre-training [93.39352617578857]
  We introduce ZimaBlue, a framework for learning generalizable World Action Models from large-scale video.<n>ZimaBlue follows a three-stage training curriculum, first performing causal embodied video pre-training on large-scale human and robot egocentric videos.<n>ZimaBlue further delivers strong performance across multiple benchmarks, with particularly pronounced gains on unseen tasks.
  arXiv  Detail & Related papers  (2026-08-31T18:09:52Z)
• VLAff: Vision-Language-Affordance Model for Unified Actionable Affordances [13.931853652256015]
  We build a framework that learns object-centric actionable affordances that are embodiment-agnostic.<n>We introduce VLAff, a vision-language model-based unified foundation model that learns cross-modal correlations across all actionable affordances.<n>VLAff achieves state-of-the-art performance on visual affordance prediction, and can be effectively applied to real robot applications.
  arXiv  Detail & Related papers  (2026-08-05T10:15:55Z)
• LUCID: Learning Embodiment-Agnostic Intent Models from Unstructured Human Videos for Scalable Dexterous Robot Skill Acquisition [11.86733592383987]
  LUCID is a framework that learns task intent from unstructured human videos.<n>It learns robot control in massively-parallel simulation.<n>We evaluate LUCID on five real-world manipulation tasks.
  arXiv  Detail & Related papers  (2026-06-10T03:49:01Z)
• Human-to-Robot Interaction: Learning from Video Demonstration for Robot Imitation [5.967530183571141]
  Human-to-Robot'' imitation learning pipeline enables robots to acquire manipulation skills directly from unstructured video demonstrations.<n>Key innovation is a modular framework that decouples the learning process into two distinct stages.<n>For robot manipulation, our framework achieves an average success rate of 87.5% across all actions, with 100% success on reaching tasks and up to 90% on complex pick-and-place operations.
  arXiv  Detail & Related papers  (2026-02-22T13:26:27Z)
• World Action Models are Zero-shot Policies [111.91938055103633]
  We introduce DreamZero, a World Action Model (WAM) built upon a pretrained video diffusion backbone.<n>By jointly modeling video and action, DreamZero learns diverse skills effectively from heterogeneous robot data.<n>We demonstrate two forms of cross-embodiment transfer: video-only demonstrations from other robots or humans yield a relative improvement of over 42% on unseen task performance.
  arXiv  Detail & Related papers  (2026-02-17T15:04:02Z)
• MiVLA: Towards Generalizable Vision-Language-Action Model with Human-Robot Mutual Imitation Pre-training [102.850162490626]
  We propose MiVLA, a vision-language-action model empowered by human-robot mutual imitation pre-training.<n>We show that MiVLA achieves strong improved generalization capability, outperforming state-of-the-art VLAs.
  arXiv  Detail & Related papers  (2025-12-17T12:59:41Z)
• H2R-Grounder: A Paired-Data-Free Paradigm for Translating Human Interaction Videos into Physically Grounded Robot Videos [58.006918399913665]
  We propose a video-to-video translation framework that converts ordinary human-object interaction videos into motion-consistent robot manipulation videos.<n>Our approach does not require any paired human-robot videos for training only a set of unpaired robot videos, making the system easy to scale.<n>At test time, we apply the same process to human videos (inpainting the person and overlaying human pose cues) and generate high-quality robot videos that mimic the human's actions.
  arXiv  Detail & Related papers  (2025-12-10T07:59:45Z)
• MimicDreamer: Aligning Human and Robot Demonstrations for Scalable VLA Training [40.45924128424013]
  We propose MimicDreamer, a framework that turns low-cost human demonstrations into robot-usable supervision.<n>For visual alignment, we propose H2R Aligner, a video diffusion model that generates high-fidelity robot demonstration videos.<n>For viewpoint stabilization, EgoStabilizer is proposed, which canonicalizes egocentric videos via homography.<n>For action alignment, we map human hand trajectories to the robot frame and apply a constrained inverse kinematics solver.
  arXiv  Detail & Related papers  (2025-09-26T11:05:10Z)
• EgoVLA: Learning Vision-Language-Action Models from Egocentric Human Videos [49.820119587446655]
  In this paper, we explore training Vision-Language-Action (VLA) models using egocentric human videos.<n>With a VLA trained on human video that predicts human wrist and hand actions, we can perform Inverse Kinematics and convert the human actions to robot actions.<n>We propose a simulation benchmark called Ego Humanoid Manipulation Benchmark, where we design diverse bimanual manipulation tasks with demonstrations.
  arXiv  Detail & Related papers  (2025-07-16T17:27:44Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.