R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies

• URL: http://arxiv.org/abs/2606.17040v1
• Date: Mon, 15 Jun 2026 17:56:19 GMT
• Title: R2RDreamer: 3D-aware Data Augmentation for Spatially-generalized 2D Manipulation Policies
• Abstract summary: R2RDreamer is a real-to-real demonstration augmentation framework.<n>It preserves the geometric consistency of 3D action-observation editing while moving visual completion to 2D video space.
• Score: 86.2249156068836
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Spatial generalization is critical for imitation-learned manipulation policies, but achieving it typically requires scaling demonstrations across diverse object poses, robot configurations, and camera viewpoints. Data augmentation from a few source demonstrations offers a practical alternative to costly real-world collection. Simulation-based augmentation can create controllable variation, but requires complex environment and object setup and may introduce a sim-to-real gap. Recent real-to-real methods avoid these issues by jointly editing 3D observations and action trajectories from real demonstrations, yet they still rely on strong 3D scene parsing and geometry completion, and often produce observations tailored to 3D pointcloud policies rather than RGB-based 2D policies. We propose R2RDreamer, a real-to-real demonstration augmentation framework that preserves the geometric consistency of 3D action-observation editing while moving visual completion to 2D video space. Specifically, R2RDreamer first performs lightweight 3D augmentation by editing incomplete object pointclouds and end-effector trajectories in a shared 3D frame; it then projects the edited scene into masked image-space control videos with occlusion-aware reasoning and uses a dense-control image-to-video model to complete temporally coherent RGB observations. Experiments on spatially shifted manipulation tasks with both 2D diffusion-style policies and vision-language-action policies show that R2RDreamer improves spatial generalization from limited source demonstrations, with analyses validating the contributions of 3D editing, occlusion-aware projection, and video completion.

Related papers

• LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models [64.11117186468533]
  World models enable agents to perform forward rollout and planning without real-world interaction.<n>Most existing LAMs rely on single-view inputs and operate primarily in 2D pixel space.<n>We propose LAWM-3D, which introduces three tightly coupled key designs for learning 3D-aware latent actions.
  arXiv  Detail & Related papers  (2026-08-06T07:47:01Z)
• Dexterity-BEV: Aligning 3D World and Actions for Generalizable Robot Policies Learning [51.799524981291235]
  End-to-end manipulation policies show promise for generalizable and dexterous robotic manipulation.<n>They inherit two key limitations from 2D foundation models.<n>We present a series of contributions to address these issues.
  arXiv  Detail & Related papers  (2026-06-01T14:01:11Z)
• 3DreamBooth: High-Fidelity 3D Subject-Driven Video Generation Model [18.407923160013404]
  We introduce a novel framework for 3D-aware video customization, comprising 3DreamBooth and 3Dapter.<n>3DreamBooth decouples spatial geometry from temporal motion through a 1-frame optimization paradigm.<n>To enhance fine-grained textures and accelerate convergence, we incorporate 3Dapter, a visual conditioning module.
  arXiv  Detail & Related papers  (2026-03-19T06:14:05Z)
• Action-Geometry Prediction with 3D Geometric Prior for Bimanual Manipulation [53.09168514034483]
  Bimanual manipulation requires policies that can reason about 3D geometry, anticipate how it evolves under action, and generate smooth, coordinated motions.<n>We propose a framework that builds bimanual manipulation directly on a pre-trained 3D geometric foundation model.<n>Our policy fuses geometry-aware latents, 2D semantic features, and proprioception into a unified state representation, and uses diffusion model to jointly predict a future action chunk and a future 3D latent that decodes into a dense pointmap.
  arXiv  Detail & Related papers  (2026-02-27T08:54:20Z)
• Real2Edit2Real: Generating Robotic Demonstrations via a 3D Control Interface [17.378883024907648]
  We present Real2Edit2Real, a framework that generates new demonstrations by bridging 3D editability with 2D visual data through a 3D control interface.<n>Our approach first reconstructs scene geometry from multi-view RGB observations with a metric-scale 3D reconstruction model.<n>We propose a multi-conditional video generation model guided by depth as the primary control signal, together with action, edge, and ray maps, to synthesize spatially augmented multi-view manipulation videos.
  arXiv  Detail & Related papers  (2025-12-22T13:53:25Z)
• Learning to Control Physically-simulated 3D Characters via Generating and Mimicking 2D Motions [23.080971732537886]
  Mimic2DM is a novel motion imitation framework that learns the control policy directly from 2D keypoint trajectories extracted from videos.<n>We show that the proposed approach is versatile and can effectively learn to synthesize physically plausible and diverse motions across a range of domains.
  arXiv  Detail & Related papers  (2025-12-09T11:30:56Z)
• Towards Physical Understanding in Video Generation: A 3D Point Regularization Approach [54.559847511280545]
  We present a novel video generation framework that integrates 3-dimensional geometry and dynamic awareness.<n>To achieve this, we augment 2D videos with 3D point trajectories and align them in pixel space.<n>The resulting 3D-aware video dataset, PointVid, is then used to fine-tune a latent diffusion model.
  arXiv  Detail & Related papers  (2025-02-05T21:49:06Z)
• PanoSLAM: Panoptic 3D Scene Reconstruction via Gaussian SLAM [105.01907579424362]
  PanoSLAM is the first SLAM system to integrate geometric reconstruction, 3D semantic segmentation, and 3D instance segmentation within a unified framework.<n>For the first time, it achieves panoptic 3D reconstruction of open-world environments directly from the RGB-D video.
  arXiv  Detail & Related papers  (2024-12-31T08:58:10Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.