OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfer

• URL: http://arxiv.org/abs/2603.14401v1
• Date: Sun, 15 Mar 2026 14:31:03 GMT
• Title: OCRA: Object-Centric Learning with 3D and Tactile Priors for Human-to-Robot Action Transfer
• Abstract summary: OCRA is a framework for video-based human-to-Robot Action transfer.<n>It learns directly from human demonstration videos to enable robust manipulation.
• Score: 84.22494391514066
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present OCRA, an Object-Centric framework for video-based human-to-Robot Action transfer that learns directly from human demonstration videos to enable robust manipulation. Object-centric learning emphasizes task-relevant objects and their interactions while filtering out irrelevant background, providing a natural and scalable way to teach robots. OCRA leverages multi-view RGB videos, the state-of-the-art 3D foundation model VGGT, and advanced detection and segmentation models to reconstruct object-centric 3D point clouds, capturing rich interactions between objects. To handle properties not easily perceived by vision alone, we incorporate tactile priors via a large-scale dataset of over one million tactile images. These 3D and tactile priors are fused through a multimodal module (ResFiLM) and fed into a Diffusion Policy to generate robust manipulation actions. Extensive experiments on both vision-only and visuo-tactile tasks show that OCRA significantly outperforms existing baselines and ablations, demonstrating its effectiveness for learning from human demonstration videos.

Related papers

• LAWM-3D: Learning 3D-Aware Latent Actions from Human Videos for Generalizable Robot World Models [64.11117186468533]
  World models enable agents to perform forward rollout and planning without real-world interaction.<n>Most existing LAMs rely on single-view inputs and operate primarily in 2D pixel space.<n>We propose LAWM-3D, which introduces three tightly coupled key designs for learning 3D-aware latent actions.
  arXiv  Detail & Related papers  (2026-08-06T07:47:01Z)
• From Human Videos to Robot Manipulation: A Survey on Scalable Vision-Language-Action Learning with Human-Centric Data [71.22409934108924]
  Human videos are abundant and capture rich interactions, providing diverse semantic and physical cues for real-world manipulation.<n>This survey provides a unified view of how human videos are transformed into effective knowledge for Vision-Language-Action (VLA) models.<n>We highlight three key open challenges in this area: structuring unstructured videos into training-ready episodes, grounding video-derived supervision into robot-executable actions under embodiment and viewpoint heterogeneity, and designing evaluation protocols that better predict real-world deployment performance and transfer efficiency.
  arXiv  Detail & Related papers  (2026-05-18T06:19:16Z)
• Learning Action Manifold with Multi-view Latent Priors for Robotic Manipulation [67.16145181790522]
  This paper tackles spatial perception and manipulation challenges in Vision-Language-Action (VLA) models.<n>We propose a Geometry-Guided Gated Transformer (G3T) that aligns multi-view features under 3D geometric guidance.<n>To improve action learning efficiency, we introduce Action Manifold Learning (AML), which directly predicts actions on the valid action manifold.
  arXiv  Detail & Related papers  (2026-05-12T09:21:29Z)
• PAWS: Perception of Articulation in the Wild at Scale from Egocentric Videos [90.4816165292854]
  Existing learning-based methods rely heavily on supervised training with high-quality 3D data and manual annotations.<n>We propose PAWS, a method that directly extracts object articulations from hand-object interactions in large-scale in-the-wild egocentric videos.<n>We evaluate our method on the public data sets, including HD-EPIC and Arti4D data sets, achieving significant improvements over baselines.
  arXiv  Detail & Related papers  (2026-03-26T15:16:51Z)
• Dexterous Manipulation Policies from RGB Human Videos via 3D Hand-Object Trajectory Reconstruction [24.49384094440561]
  We propose VIDEOMANIP, a device-free framework that learns dexterous manipulation directly from RGB human videos.<n>In simulation, the learned grasping model achieves a 70.25% success rate across 20 diverse objects using the Inspire Hand.<n>In the real world, manipulation policies trained from RGB videos achieve an average 62.86% success rate across seven tasks using the LEAP Hand.
  arXiv  Detail & Related papers  (2026-02-09T18:56:02Z)
• DynaRend: Learning 3D Dynamics via Masked Future Rendering for Robotic Manipulation [52.136378691610524]
  We present DynaRend, a representation learning framework that learns 3D-aware and dynamics-informed triplane features.<n>By pretraining on multi-view RGB-D video data, DynaRend jointly captures spatial geometry, future dynamics, and task semantics in a unified triplane representation.<n>We evaluate DynaRend on two challenging benchmarks, RLBench and Colosseum, demonstrating substantial improvements in policy success rate, generalization to environmental perturbations, and real-world applicability across diverse manipulation tasks.
  arXiv  Detail & Related papers  (2025-10-28T10:17:11Z)
• O$^3$Afford: One-Shot 3D Object-to-Object Affordance Grounding for Generalizable Robotic Manipulation [8.1159855043566]
  We address the challenge of object-to-object affordance grounding under limited data contraints.<n>Inspired by recent advances in few-shot learning with 2D vision foundation models, we propose a novel one-shot 3D object-to-object affordance learning approach for robotic manipulation.<n>Our experiments on 3D object-to-object affordance grounding and robotic manipulation demonstrate that our O$3$Afford significantly outperforms existing baselines in terms of both accuracy and generalization capability.
  arXiv  Detail & Related papers  (2025-09-07T22:45:06Z)
• Learning Video Generation for Robotic Manipulation with Collaborative Trajectory Control [72.00655365269]
  We present RoboMaster, a novel framework that models inter-object dynamics through a collaborative trajectory formulation.<n>Unlike prior methods that decompose objects, our core is to decompose the interaction process into three sub-stages: pre-interaction, interaction, and post-interaction.<n>Our method outperforms existing approaches, establishing new state-of-the-art performance in trajectory-controlled video generation for robotic manipulation.
  arXiv  Detail & Related papers  (2025-06-02T17:57:06Z)
• TASTE-Rob: Advancing Video Generation of Task-Oriented Hand-Object Interaction for Generalizable Robotic Manipulation [18.083105886634115]
  TASTE-Rob is a dataset of 100,856 ego-centric hand-object interaction videos.<n>Each video is meticulously aligned with language instructions and recorded from a consistent camera viewpoint.<n>To enhance realism, we introduce a three-stage pose-refinement pipeline.
  arXiv  Detail & Related papers  (2025-03-14T14:09:31Z)
• VidBot: Learning Generalizable 3D Actions from In-the-Wild 2D Human Videos for Zero-Shot Robotic Manipulation [53.63540587160549]
  VidBot is a framework enabling zero-shot robotic manipulation using learned 3D affordance from in-the-wild monocular RGB-only human videos.<n> VidBot paves the way for leveraging everyday human videos to make robot learning more scalable.
  arXiv  Detail & Related papers  (2025-03-10T10:04:58Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.