EMPIRE: Explicit Manipulation Planning as a Learnable Intermediate Representation for Egocentric Hand-Motion Forecasting
Abstract Overview
EMPIRE is a two-stage framework for egocentric bimanual hand-motion forecasting that inserts an explicit manipulation plan between multimodal observation and motion generation. In Stage I, a vision-language planner predicts temporally ordered, hand-specific manipulation steps from RGB, monocular depth, a coarse caption, and camera field of view. In Stage II, a flow-matching diffusion transformer generates future hand motion while conditioning on the frozen planner's hidden states, preventing motion-training gradients from altering the planning representation. The paper also introduces EMPIRE-651K, a dataset of 650,910 five-second forecasting windows across 111 manipulation tasks, each paired with explicit per-hand manipulation plans.
Novelty
The paper introduces explicit manipulation planning as a learnable intermediate representation for egocentric hand-motion forecasting rather than mapping visual-linguistic observations directly to motion. It couples this with a decoupled plan-then-act training scheme where the planner is frozen during motion generator training, supported by a newly constructed large-scale dataset with plan supervision.
Results
On EMPIRE-651K, EMPIRE achieves state-of-the-art forecasting accuracy with an MPJPE of 84.53 mm and a finger-relative MPJPE of 38.97 mm. It reduces MPJPE by 19.8% relative to the re-implemented VITRA baseline while decreasing training time by 38.8%, and attains comparable overall accuracy to Being-H0-14B with 83.5× faster inference. Task-level analysis further shows that explicit planning provides greater benefit on more complex tasks, improving performance on 89 of 111 evaluated manipulation tasks.
Key Points
- EMPIRE decouples forecasting into two stages: a VLM generates structured per-hand manipulation plans from multimodal inputs, and a flow-matching DiT actor forecasts future bimanual motion conditioned on the frozen planner's hidden states.
- The authors construct EMPIRE-651K from EgoDex, providing 650,910 training windows across 111 manipulation tasks with MANO-based target trajectories and VLM-generated, audit-filtered manipulation plans.
- Experimental results and ablation studies demonstrate that explicit plan conditioning and isolating motion gradients from the VLM substantially improve fine-grained finger articulation, overall forecasting accuracy, training efficiency, and cross-task generalization.