RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation

• URL: http://arxiv.org/abs/2607.06558v2
• Date: Sun, 12 Jul 2026 13:53:14 GMT
• Title: RynnWorld-Teleop: An Action-Conditioned World Model for Digital Teleoperation
• Abstract summary: We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing real robot with a generative world model.<n>We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation.<n>This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU.
• Score: 48.828787848144806
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Scaling robot learning requires massive, diverse trajectory data, yet collection is currently bottlenecked by physical teleoperation, where every demonstration binds operator time to specific hardware and workspaces. We introduce digital teleoperation, a paradigm that decouples data collection from physical constraints by replacing the real robot with a generative world model. In this framework, an operator's hand-pose stream drives a robot-centric generative world model to synthesize high-fidelity egocentric videos from a single reference image. The recorded pose stream serves as an embodiment-agnostic action label transferable to any target robot via standard retargeting, yielding complete state-action trajectories for imitation learning independent of physical hardware. We instantiate this paradigm in RynnWorld-Teleop, a system that integrates depth-aware skeletal conditioning, progressive human-to-robot training on a video Diffusion Transformer, and streaming autoregressive distillation. This pipeline compresses the generative process into a single-pass inference, enabling 40+ FPS, real-time interactive generation on a single H100 GPU. Policies trained exclusively on RynnWorld-Teleop-generated data achieve effective zero-shot Sim2Real transfer across dexterous and diverse bimanual tasks. Moreover, augmenting real-world datasets with our digitally teleoperated data consistently improves success rates, demonstrating that RynnWorld-Teleop serves as a high-fidelity, scalable data engine for the next generation of robotic agents.

Related papers

• AnyWorld: Factorized Egocentric World Models for Cross-Embodiment Generalization [88.25615187835321]
  We propose AnyWorld, a cross-embodiment world modeling framework.<n>Our model factorizes an interaction into action, camera, and embodiment.<n>We train the model with large-scale human interaction pretraining.
  arXiv  Detail & Related papers  (2026-08-29T12:55:15Z)
• H2R-Bench: Benchmarking Human-to-Robot Manipulation Video Generation in World Models [56.60098789696351]
  H2R-Bench is a benchmark for evaluating cross-embodiment human-to-robot manipulation video generation.<n>Each benchmark instance contains a human demonstration video, target embodiment constraints, and source-grounded annotations.<n>We benchmark eleven state-of-the-art video generation models across six manipulation families and two robot embodiments.
  arXiv  Detail & Related papers  (2026-08-13T10:14:33Z)
• Ego2Robot: Scalable Robot Data Synthesis from Egocentric Human Data [60.94418315274988]
  Ego2Robot supports both curated and in-the-wild videos, producing 18,561 hours of robot training data spanning 15 robot morphologies.<n>Joint pretraining on Ego2Robot-synthesized and robot data consistently improves out-of-distribution generalization across multiple types.
  arXiv  Detail & Related papers  (2026-08-03T17:52:26Z)
• EgoInfinity: A Web-Scale 4D Hand-Object Interaction Data Engine for Any-View Robot Retargeting and Video-to-Action Robot Learning [10.780924973366737]
  EgoInfinity is a universal 4D hand-object interaction data engine that enables web-scale data generation for robot acquisition and learning.<n>EgoInfinity is a modular engine integrating perception, segmentation, reconstruction, interaction-aware refinement, and naive to automate this traditionally unscalable video-to-action problem.
  arXiv  Detail & Related papers  (2026-06-16T00:44:16Z)
• PlayWorld: Learning Robot World Models from Autonomous Play [8.710835027281465]
  We present PlayWorld, a simple, scalable, and fully autonomous pipeline for training high-fidelity video world simulators from interaction experience.<n>PlayWorld is the first system capable of learning entirely from unsupervised robot self-play, enabling naturally scalable data collection.<n> Experiments show that PlayWorld generates high-quality, physically consistent predictions for contact-rich interactions that are not captured by world models trained on human-collected data.
  arXiv  Detail & Related papers  (2026-03-09T23:58:07Z)
• DreamDojo: A Generalist Robot World Model from Large-Scale Human Videos [110.98100817695307]
  We introduce DreamDojo, a foundation world model that learns diverse interactions and dexterous controls from 44k hours of egocentric human videos.<n>Our work enables several important applications based on generative world models, including live teleoperation, policy evaluation, and model-based planning.
  arXiv  Detail & Related papers  (2026-02-06T18:49:43Z)
• AnchorDream: Repurposing Video Diffusion for Embodiment-Aware Robot Data Synthesis [33.90053396451562]
  We introduce AnchorDream, an embodiment-aware world model that repurposes pretrained video diffusion models for robot data synthesis.<n>Our method scales them into large, diverse, high-quality datasets without requiring explicit environment modeling.<n>Experiments show that the generated data leads to consistent improvements in downstream policy learning, with relative gains of 36.4% in simulator benchmarks and nearly double performance in real-world studies.
  arXiv  Detail & Related papers  (2025-12-12T18:59:45Z)
• RobotSeg: A Model and Dataset for Segmenting Robots in Image and Video [56.9581053843815]
  We introduce RobotSeg, a foundation model for robot segmentation in image and video.<n>It addresses the lack of adaptation to articulated robots, reliance on manual prompts, and the need for per-frame training mask annotations.<n>It achieves state-of-the-art performance on both images and videos.
  arXiv  Detail & Related papers  (2025-11-28T07:51:02Z)
• ORV: 4D Occupancy-centric Robot Video Generation [33.360345403049685]
  Acquiring real-world robotic simulation data through teleoperation is notoriously time-consuming and labor-intensive.<n>We propose ORV, an Occupancy-centric Robot Video generation framework, which utilizes 4D semantic occupancy sequences as a fine-grained representation.<n>By leveraging occupancy-based representations, ORV enables seamless translation of simulation data into photorealistic robot videos, while ensuring high temporal consistency and precise controllability.
  arXiv  Detail & Related papers  (2025-06-03T17:00:32Z)
• DreamGen: Unlocking Generalization in Robot Learning through Video World Models [120.25799361925387]
  DreamGen is a pipeline for training robot policies that generalize across behaviors and environments through neural trajectories.<n>Our work establishes a promising new axis for scaling robot learning well beyond manual data collection.
  arXiv  Detail & Related papers  (2025-05-19T04:55:39Z)
• IRASim: A Fine-Grained World Model for Robot Manipulation [24.591694756757278]
  We present IRASim, a novel world model capable of generating videos with fine-grained robot-object interaction details.<n>We train a diffusion transformer and introduce a novel frame-level action-conditioning module within each transformer block to explicitly model and strengthen the action-frame alignment.
  arXiv  Detail & Related papers  (2024-06-20T17:50:16Z)
• RT-1: Robotics Transformer for Real-World Control at Scale [98.09428483862165]
  We present a model class, dubbed Robotics Transformer, that exhibits promising scalable model properties. We verify our conclusions in a study of different model classes and their ability to generalize as a function of the data size, model size, and data diversity based on a large-scale data collection on real robots performing real-world tasks.
  arXiv  Detail & Related papers  (2022-12-13T18:55:15Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.