Daily Paper Briefing

Abstract Overview

This paper presents PRISM, a real-to-sim-to-real framework for teaching humanoid loco-manipulation from a small number of real human videos. The method uses video-to-video generation to expand four seed recordings into 256 counterfactual human-object interaction videos, then reconstructs human and object motions and retargets them into physically plausible robot demonstrations using contact-anchored constraints. A unified policy is trained from these trajectories through a privileged teacher and a depth-based student that uses onboard depth and joystick commands. The paper evaluates reconstruction quality, sim-to-sim transfer, real-world deployment, and ablations to test the contribution of the generated data and contact-aware components.

Novelty

The distinctive idea is to use counterfactual video generation as a scalable source of human-object interaction data for humanoid robot learning, rather than relying only on collected demonstrations or direct robot video generation. The paper also introduces a contact-anchored reconstruction and retargeting pipeline that uses inferred contact structure to convert noisy monocular video reconstructions into usable robot-object trajectories.

Results

The authors deploy the learned policy zero-shot on a real Unitree G1 without real-world fine-tuning, where it performs pick-up, carry, and drop behaviors using only onboard depth observations and joystick commands. In simulation, PRISM-trained policies transfer better across domains than policies trained only on OMOMO data, and the full contact-anchored system improves success from 22.50%/12.50% (ID/OOD) in the baseline to 96.25%/72.92%. Real-world evaluation reports successful trials across in-domain and several out-of-domain objects, including 15/15 for boxes and 14/15 for bins and barrels.

Key Points

1. PRISM expands four real videos into 256 counterfactual interaction clips and uses them to train a single humanoid loco-manipulation policy.
2. The method relies on contact-anchored object reconstruction, retargeting, and reward design to turn imperfect monocular video data into physically plausible training trajectories.
3. The resulting depth-based policy transfers zero-shot to a real humanoid and generalizes across object instances, poses, scales, and some out-of-domain categories.

References

• arXiv: https://arxiv.org/abs/2609.38172v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2609.38172v1
• Hugging Face Papers: https://huggingface.co/papers/2609.38172
• Project: https://prism-real2sim2real.github.io/