TempAct: Advancing Temporal Plausibility in Autoregressive Video Generation via Planner-Executor RL
Abstract Overview
TempAct addresses a weakness of autoregressive video generation for streaming synthesis: a single global prompt does not specify which sub-event should occur in each generated chunk, while naive step-wise prompting can cause delayed reactions, blended semantics, and error propagation at prompt transitions. The method introduces a planner-executor reinforcement learning framework in which an LLM planner decomposes a global instruction into span-aware step prompts and an autoregressive video diffusion executor generates chunks conditioned on these steps and prior visual context. The two components are trained jointly with hierarchical group exploration and separate planner- and executor-level rewards, including temporal-following, plan-quality, local step-following, and aesthetic signals. Experiments on temporally structured prompts evaluate whether generated videos preserve event order and state transitions, rather than only generic visual quality.
Novelty
The paper's main novelty is to cast temporally plausible autoregressive video generation as a joint planning-and-execution problem rather than only an executor alignment problem. It combines an LLM planner, a diffusion-based executor, hierarchical planner/executor RL, and transition-aware prompt smoothing so that temporal decomposition and prompt-switch execution are optimized together.
Results
On the Temporal Order benchmark, TempAct improves Temporal-Following Score over step-prompt baselines on both Self-Forcing and LongLive backbones while largely preserving VBench and PickScore. For Self-Forcing, the average score rises from 0.400 to 0.462, and for LongLive it rises from 0.432 to 0.488; gains appear under both Qwen3-VL and Gemini judges. User studies also report that TempAct is preferred more often than the corresponding baselines, and evaluator-human agreement is reported at 76% for Qwen3-VL and 81% for Gemini-3-Flash.
Key Points
- TempAct uses an LLM planner to sample span-aware step decompositions of a global prompt and an autoregressive diffusion executor to realize them under generated visual history.
- Its training uses hierarchical reinforcement learning with separate credit assignment: plan-level rewards guide executable temporal decompositions, while local transition-level rewards improve prompt-switch execution and reduce error propagation.
- The method improves temporal instruction following across two autoregressive video backbones without materially degrading overall visual quality metrics, and the improvements are supported by both VLM-based evaluation and human preference.