In-Context Model Predictive Generation: Open-Vocabulary Motion Synthesis from Language Models to Physics
Abstract Overview
This paper introduces In-Context Model Predictive Generation (ICMPG), a framework for text-driven human motion synthesis that combines large language model planning with inference-time physical feedback. The method decomposes open-vocabulary instructions into structured action descriptions, generates candidate motion-token sequences, and then iteratively refines them using a model-predictive procedure guided by semantic and physical rewards. Its architecture includes a Context-Aware Motion Generation module for LLM-based token generation and a Model Predictive Generation module that evaluates candidates in simulation and selects the best continuation. The goal is to reduce the usual trade-off between semantic expressiveness and physical plausibility in open-vocabulary motion generation.
Novelty
The distinctive contribution is the reformulation of motion synthesis as an MPC-like closed-loop generation process rather than open-loop text-to-motion decoding or task-specific reinforcement learning. The framework uses frozen or lightly adapted LLMs as planners/generators and combines them with simulation-based physical feedback at inference time, avoiding task-specific policy retraining while retaining open-vocabulary flexibility.
Results
On the open-vocabulary BABEL benchmark, the LoRA-tuned variant achieves the best reported semantic metrics among the compared methods, including FID 0.492, CLIP_Score 29.32, and R-Precision 0.316/0.603, while maintaining 82.9% simulation success and Phys-Err 3.704. The non-fine-tuned ICMPG also outperforms AvatarCLIP and MotionCLIP on FID, CLIP_Score, and R-Precision, and the paper reports competitive results on HumanML3D and KIT-ML with improved physically constrained realism relative to non-physics-aware baselines.
Key Points
- ICMPG couples LLM-based semantic planning with simulation-based reward feedback, selecting motion continuations through a receding-horizon refinement loop.
- The method targets open-vocabulary motion synthesis and uses SemAligner, motion tokenization via VQ-VAE, and combined semantic/physical rewards to improve alignment and feasibility.
- Experiments show stronger open-vocabulary generalization and competitive physical realism, with ablations indicating that the planner, reward design, and LLM backbone materially affect performance.