Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models

• URL: http://arxiv.org/abs/2605.18601v1
• Date: Mon, 18 May 2026 16:12:52 GMT
• Title: Incantation: Natural Language as the Action Interface for Multi-Entity Video World Models
• Abstract summary: Incantation is the first interactive video world model with per-latent-frame natural-language conditioning.<n>We surpass the Action-Index baseline on cross-entity transfer and out-of-vocabulary prompts.<n>We apply the same architecture and training recipe to The King of Fighters, changing only the per-entity action vocabulary slots.
• Score: 18.614835778452345
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern interactive video world models have achieved impressive visual fidelity, yet lack fine-grained multi-entity control and cross-entity, cross-world generalization. We trace this gap to the action interface: standard control protocols (e.g. animation IDs, device inputs, scene-level captions) bind action semantics to specific entities or engines at design time. We propose natural language as the interface to unlock expressiveness that no prior interface can achieve, and we present Incantation, the first interactive video world model with per-latent-frame (0.25 s) natural-language conditioning that supports simultaneous multi-entity control and concept-level cross-entity transfer beyond any fixed rendering pipeline. We pair a pretrained bidirectional video backbone with frame-local text cross-attention, and enable real-time long-horizon streaming through ODE-initialized Self-Forcing distillation with a RoPE-decoupled sliding KV-cache. We surpass the Action-Index baseline on cross-entity transfer (89% vs. 43%) and out-of-vocabulary prompts (90% vs. 0%), and our 2-step student sustains 19.7 FPS at 480p with stable FVD over 2-hour rollouts. We further apply the same architecture and training recipe to The King of Fighters, changing only the per-entity action vocabulary slots. We have released a preview subset of the Incantation dataset at https://huggingface.co/datasets/zhush/incantation-elden-ring-scenes, containing manually collected Elden Ring player-boss combat clips with structured action-oriented metadata. Larger-scale Elden Ring and KOF data will be released with the full project.

Related papers

• ABot-World-0: Infinite Interactive World Rollout on a Single Desktop GPU [41.37131555577374]
  ABot-World-0 is an action-conditioned video world model for real-time, long-horizon closed-loop interaction.<n>WorldExplorer performs agent-driven collection guided by training feedback.<n> unified pipeline applies 14 deterministic quality checks, VLM-based assessment, and synchronized action and text annotation.
  arXiv  Detail & Related papers  (2026-07-21T15:26:50Z)
• CineOrchestra: Unified Entity-Centric Conditioning for Cinematic Video Generation [57.47993858752813]
  We present CineOrchestra, a unified video diffusion model that controls subjects, events, cameras, and shot transitions simultaneously.<n>Our key insight is that these heterogeneous cinematic elements share a fundamental structure.<n>CineOrchestra outperforms six per-axis specialists on dense caption following and shot-transition timing.
  arXiv  Detail & Related papers  (2026-06-11T17:59:10Z)
• SCAIL-2: Unifying Controlled Character Animation with End-to-end In-Context Conditioning [12.196865711049561]
  Controlled character animation requires transferring motion from a driving sequence to a reference character.<n>We present SCAIL-2, a framework that bypasses those intermediates and achieves textbfend-to-end character animation.
  arXiv  Detail & Related papers  (2026-06-09T12:49:34Z)
• CLAW: Composable Language-Annotated Whole-body Motion Generation [55.99805728566105]
  We present CLAW, a pipeline for scalable generation of language-annotated whole-body motion data.<n>CLAW composes motion primitives from a kinematic planner, parameterized by movement, heading, speed, pelvis height, and duration.<n>A low-level controller tracks these references in MuJoCo simulation, yielding physically grounded trajectories.
  arXiv  Detail & Related papers  (2026-04-13T10:02:04Z)
• Olaf-World: Orienting Latent Actions for Video World Modeling [100.96069208914957]
  Scaling action-controllable world models is limited by the scarcity of action labels.<n>We present Olaf-World, a pipeline that pretrains action-conditioned video world models from large-scale passive video.
  arXiv  Detail & Related papers  (2026-02-10T18:58:41Z)
• VideoMolmo: Spatio-Temporal Grounding Meets Pointing [66.19964563104385]
  VideoMolmo is a model tailored for fine-grained pointing of video sequences.<n>A novel temporal mask fusion employs SAM2 for bidirectional point propagation.<n>To evaluate the generalization of VideoMolmo, we introduce VPoMolS-temporal, a challenging out-of-distribution benchmark spanning five real-world scenarios.
  arXiv  Detail & Related papers  (2025-06-05T17:59:29Z)
• TANGO: Co-Speech Gesture Video Reenactment with Hierarchical Audio Motion Embedding and Diffusion Interpolation [4.019144083959918]
  We present TANGO, a framework for generating co-speech body-gesture videos. Given a few-minute, single-speaker reference video, TANGO produces high-fidelity videos with synchronized body gestures.
  arXiv  Detail & Related papers  (2024-10-05T16:30:46Z)
• Summarize the Past to Predict the Future: Natural Language Descriptions of Context Boost Multimodal Object Interaction Anticipation [72.74191015833397]
  We propose TransFusion, a multimodal transformer-based architecture. It exploits the representational power of language by summarizing the action context. Our model enables more efficient end-to-end learning.
  arXiv  Detail & Related papers  (2023-01-22T21:30:12Z)
• Adaptive Perception Transformer for Temporal Action Localization [13.735402329482719]
  This paper proposes a novel end-to-end model, called adaptive perception transformer (AdaPerFormer) One branch takes care of the global perception attention, which can model entire video sequences and aggregate global relevant contexts. The other branch concentrates on the local convolutional shift to aggregate intra-frame and inter-frame information.
  arXiv  Detail & Related papers  (2022-08-25T07:42:48Z)
• HERO: Hierarchical Encoder for Video+Language Omni-representation Pre-training [75.55823420847759]
  We present HERO, a novel framework for large-scale video+language omni-representation learning. HERO encodes multimodal inputs in a hierarchical structure, where local context of a video frame is captured by a Cross-modal Transformer. HERO is jointly trained on HowTo100M and large-scale TV datasets to gain deep understanding of complex social dynamics with multi-character interactions.
  arXiv  Detail & Related papers  (2020-05-01T03:49:26Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.