Marionette: Predicting World States, Rendering Geometry, Painting Appearance

• URL: http://arxiv.org/abs/2608.14530v1
• Date: Fri, 14 Aug 2026 17:48:16 GMT
• Title: Marionette: Predicting World States, Rendering Geometry, Painting Appearance
• Abstract summary: We instantiate this idea as Marionette, a world model for interactive games with articulated characters.<n>A two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state.<n>A control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls.
• Score: 30.170848813370444
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Interactive game world models typically autoregress visual observations directly in pixel or latent space, forcing structured properties such as pose, geometry, and occlusion to be implicitly maintained by the same generative sequence. Over long horizons, errors in these latent world properties accumulate, making consistency and controllability fragile. We explicitly model the evolving world state, delegate exact geometric computation to a fixed, zero-parameter renderer, and leave the neural model to synthesize appearance. We instantiate this idea as Marionette, a world model for interactive games with articulated characters. First, a two-stage autoregressive dynamics model predicts an explicit and interpretable 276-dimensional 3D world state comprising multi-entity articulated skeletons, metric root trajectories, and rotations. Second, a zero-parameter graphics bridge converts the predicted state into pose-control videos, computing world-space geometry and occlusion in closed form. Third, a control-conditioned video-diffusion observation model synthesizes photorealistic RGB observations from the resulting structured controls. Our experiments establish two properties of Marionette. First, the predicted world state is directly controllable. Forcing a mismatched action stream changes root-aligned joint error by 31% across 48 held-out segments. Second, long-horizon behaviour is determined in the state, and can be repaired there. Left free, the two generated characters drift to 21.2 m apart (recorded sessions stay near 5 m) and a third of frames show ground penetration. Two rules imposed on the explicit state, a terrain collider and a separation cap, cut penetration by 66% and keep the pair engaged, with no change to the observation model. Routing appearance through the predicted state costs no fidelity we can detect, at an FVD of 831 against 799 for recorded pose.

Related papers

• Programmable World Model [37.43654073749753]
  We introduce Programmable World Model, a framework that decouples world-state evolution from visual observation generation.<n>An agent translates natural-language instructions into executable programs that specify entity states and state-transition rules.<n>A lightweight engine executes these programs to update and maintain an explicit, persistent global world state.
  arXiv  Detail & Related papers  (2026-09-09T17:59:32Z)
• Can Video World Models Track Unobserved World States? [40.869875403179535]
  Video world models are increasingly used as simulators, yet visual fidelity alone does not show that a model maintains the hidden state of the world.<n>We examine this gap with an action-conditioned video Shell Game, a visual analog of $S_5$ state tracking.<n>We find two mechanisms that do extrapolate, and both carry a state across chunks and revise it in place.
  arXiv  Detail & Related papers  (2026-08-31T12:29:24Z)
• 4DGS-WAM: Bridging Past and Future with an Object-Centric World Action Model based on 4D Gaussian Splatting [72.02343855552051]
  Current world action models (WAMs) typically operate on 2D visual data.<n>We leverage an explicit 4D Gaussian Splatting (4DGS) representation that separately models dynamic objects and the static background of a scene.<n>4DGS-WAM lifts 2D observations into a persistent 4D representation so that previously observed static content can be reused during future prediction.
  arXiv  Detail & Related papers  (2026-08-26T16:16:57Z)
• MASS: Multiplayer World Models with Authoritative Shared State [65.85526248184442]
  MASS (Multiplayer world models with Authoritative Shared State) is inspired by multiplayer game architectures.<n>A learned Logic Engine advances a global, authoritative typed state from joint actions.<n>A learned Rendering Engine generates independent and consistent views for any requested camera on demand.
  arXiv  Detail & Related papers  (2026-08-06T16:47:24Z)
• StatePlay: State-Aware Game World Models for Mechanics-Consistent Generation [53.07918503798731]
  StatePlay is a novel state-aware game world model that predicts visual content and game states to promote mechanics-consistent generation.<n>Our work highlights the importance of state-aware game world modeling and advances beyond pixel-level realism toward complete and mechanically faithful game generation.
  arXiv  Detail & Related papers  (2026-07-29T10:49:30Z)
• Deform360: A Massive Multi-view Visuotactile Dataset for Deformable World Models [36.26785019314161]
  Deform360 is a large-scale visuotactile dataset featuring 198 daily-life objects, 1,980 interaction sequences, and over 215 hours of observations from 41 surround-view cameras and bimanual tactiles.<n>We evaluate current state-of-the-art world models, comparing 2D video models against 3D particle models.
  arXiv  Detail & Related papers  (2026-07-06T17:59:18Z)
• Sensorimotor World Models: Perception for Action via Inverse Dynamics [59.121736411156384]
  We introduce a sensorimotor world model trained end-to-end with inverse dynamics regularization.<n>It prevents representation collapse and induces action-aligned representations.<n>It learns compact, interpretable latent spaces and enables competitive planning performance across simple 2D and 3D control tasks.
  arXiv  Detail & Related papers  (2026-06-18T11:25:16Z)
• RAYNOVA: Scale-Temporal Autoregressive World Modeling in Ray Space [51.441415833480505]
  RAYNOVA is a multiview world model for driving scenarios that employs a dual-causal autoregressive framework.<n>It constructs an isotropic-temporal representation across views, frames, and scales based on relative Plcker-ray positional encoding.
  arXiv  Detail & Related papers  (2026-02-24T08:41:40Z)
• Walk through Paintings: Egocentric World Models from Internet Priors [65.30611174953958]
  We present the Egocentric World Model (EgoWM), a simple, architecture-agnostic method that transforms any pretrained video diffusion model into an action-conditioned world model.<n>Rather than training from scratch, we repurpose the rich world priors of Internet-scale video models and inject motor commands through lightweight conditioning layers.<n>Our approach scales naturally across embodiments and action spaces, ranging from 3-DoF mobile robots to 25-DoF humanoids.
  arXiv  Detail & Related papers  (2026-01-21T18:59:32Z)
• Towards High-Consistency Embodied World Model with Multi-View Trajectory Videos [24.111891848073288]
  Embodied world models aim to predict and interact with the physical world through visual observations and actions.<n>MTV-World introduces Multi-view Trajectory-Video control for precise visuomotor prediction.<n>MTV-World achieves precise control execution and accurate physical interaction modeling in complex dual-arm scenarios.
  arXiv  Detail & Related papers  (2025-11-17T02:17:04Z)
• DSG-World: Learning a 3D Gaussian World Model from Dual State Videos [14.213608866611784]
  We present DSG-World, a novel end-to-end framework that explicitly constructs a 3D Gaussian World model from Dual State observations.<n>Our approach builds dual segmentation-aware Gaussian fields and enforces bidirectional photometric and semantic consistency.
  arXiv  Detail & Related papers  (2025-06-05T16:33:32Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.