Daily Paper Briefing

Abstract Overview

RepWAM is a representation-centric world action model designed for robot manipulation. The method replaces reconstruction-oriented video latents with a semantic visual-action tokenization pipeline that aligns visual latents to a frozen visual foundation model and learns latent action tokens as transitions between visual states. On top of these aligned tokens, the model pretrains a causal diffusion transformer to jointly predict future visual states and latent actions under language instructions, then adapts to real robot demonstrations for closed-loop control. The paper evaluates the approach on real-world Franka dual-arm tasks, RoboTwin 2.0 simulation, and several ablations on tokenizer and latent-action design.

Novelty

The main novelty is the representation visual-action tokenizer, which jointly builds semantically aligned visual tokens and latent action tokens in a shared space rather than relying on reconstruction-focused video tokenizers and separate action spaces. The work also uses a two-stage world action modeling setup that pretrains on future visual states and latent actions before adapting to embodiment-specific robot control.

Results

In real-world experiments, RepWAM-5B achieved 60% success on fruit picking, 80% on drawer pushing, and 60% on test-tube insertion over 10 rollouts per task, outperforming the reported baselines on the latter two tasks and tying the best result on the first. On RoboTwin 2.0, the 5B model reached 89.3 on Easy tasks and 88.4 on Hard tasks, while ablations showed that the proposed semantic tokenizer and two-stage latent-action training improved visual dynamics metrics, open-loop action scores, and closed-loop manipulation success over reconstruction-oriented or joint-prediction alternatives.

Key Points

1. RepWAM aligns video latents with a frozen visual foundation model and learns latent actions as semantic transitions between visual states.
2. The world action model jointly generates future visual states and latent actions under language conditioning, then is adapted to robot demonstrations for control.
3. Experiments and ablations indicate gains on real-robot manipulation, RoboTwin 2.0, and tokenizer/action-modeling metrics relative to the compared alternatives.

References

• arXiv: https://arxiv.org/abs/2606.13674v1
• Fugu-MT: https://fugumt.com/fugumt/paper_check/2606.13674v1
• Hugging Face Papers: https://huggingface.co/papers/2606.13674
• GitHub: https://github.com/wdrink/RepWAM
• Project: https://wdrink.github.io/RepWAM