Learning Skills from Historical Action Trajectories: Action Experience Dictionary for World Action Models
Abstract Overview
This paper proposes an Action Experience Dictionary (AED) for world action models in robotic manipulation. The method encodes historical action trajectories into a shared dictionary of action embeddings, retrieves task-relevant entries via a pretrained action tokenizer, and then visually conditions the retrieved embeddings before using them to guide action prediction. The model also introduces a motion-aware transition loss that predicts visual feature changes over random temporal intervals to emphasize action-related motion and reduce reliance on background cues. The approach is evaluated on simulation benchmarks including LIBERO, RoboTwin 2.0, and LIBERO-Plus, as well as real-world cross-embodiment settings.
Novelty
The distinctive contribution is an explicit mechanism for reusing historical action experience across manipulation tasks within a world action model. Rather than only learning task-specific behaviors, the paper uses a shared action dictionary plus visual conditioning and a motion-aware transition objective to model cross-task skill relationships and action-centric motion.
Results
The reported experiments show the best average success rates among compared methods on RoboTwin 2.0 (92.8%) and LIBERO (98.8%), and an 86.2% overall success rate on LIBERO-Plus without embodied policy pretraining. Ablations on LIBERO-10 indicate that removing the motion-aware loss, visual conditioning, or the AED reduces performance, and real-world tests on two robot platforms show stronger robustness than baselines under clutter, low light, and unseen-object conditions.
Key Points
- AED stores shared action embeddings derived from historical trajectories, enabling reuse of manipulation skill patterns across related tasks.
- The method visually conditions retrieved action embeddings and prepends them to action tokens so prediction can use both action experience and task-relevant scene context.
- A motion-aware transition loss supervises predicted visual feature changes over random intervals, and empirical studies link this design to improved benchmark performance and OOD robustness.