WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models
- URL: http://arxiv.org/abs/2605.25077v1
- Date: Sun, 24 May 2026 13:40:42 GMT
- Title: WorldCraft: From Camera Navigation to Object Manipulation in Interactive Video World Models
- Abstract summary: We present WorldCraft, a framework that expands interactive video world models from camera navigation to object-level trajectory actions.<n>Given a user click and a sketched path, WorldCraft generates future frames in which the selected object follows the prescribed trajectory.<n>Experiments show that WorldCraft enables accurate object control, preserves the video-based world model's camera fidelity under camera-only evaluation, and maintains object state across long autoregressive rollouts.
- Score: 32.26032900976109
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: Recent video-based world models have made pixel-space environments interactive at the camera level: users can navigate viewpoints while the model generates coherent visual continuations. Yet their action spaces remain incomplete: users can move the camera, but cannot act on individual objects. Since real-world interaction is inherently object-centric, such models remain closer to passive scene observers than truly manipulable environments. We present WorldCraft, a framework that expands interactive video world models from camera navigation to object-level trajectory actions. Given a user click and a sketched path, WorldCraft generates future frames in which the selected object follows the prescribed trajectory while the camera continues to navigate the scene. WorldCraft achieves this through a trajectory-centric control pipeline: First, Normalized World Trajectory (NWT) represents user-drawn motion in a camera-invariant world coordinate system and dynamically re-projects it under the current camera pose, separating object motion from camera-induced screen-space displacement; Spatial-Pathway LoRA (SP-LoRA) then injects this world-space signal through the model's spatial-control pathway, adding object manipulation capability while preserving the pretrained camera controller; finally, Trajectory-Anchored State Persistence (TASP) treats the world trajectory as a persistent spatial state and refreshes autoregressive memory after trajectory-conditioned generation, allowing moved objects to reappear at their updated positions after leaving the camera view. Experiments show that WorldCraft enables accurate object control, preserves the video-based world model's camera fidelity under camera-only evaluation, and maintains object state across long autoregressive rollouts with off-camera excursions.
Related papers
- UniMoCa: Unifying Motion and Camera Controls as Visual Proxies for Faithful Human Video Generation [50.391925716274926]
textbfUniMoCa is a representation-driven framework that unifies motion and camera controls in visual space.<n>We show that UniMoCa achieves substantial gains in human motion control, camera control, temporal consistency, and camera-aware robustness with minimal additional complexity.
arXiv Detail & Related papers (2026-08-03T09:18:13Z) - WorldDirector: Building Controllable World Simulators with Persistent Dynamic Memory [106.28234880163723]
WorldDirector is a highly controllable video world model framework.<n>Our framework explicitly decouples semantic motion orchestration from visual generation.<n>We successfully preserve the exact visual identities of dynamic entities even when they re-enter the scene after prolonged periods out of view.
arXiv Detail & Related papers (2026-07-02T17:59:59Z) - Directing the World: Fast Autoregressive Video Generation with Compositional Human-Camera Control [56.828569670698975]
"Directing the World" is a fast autoregressive framework for controllable world-model video generation with compositional human-motion and camera-trajectory control.<n>Our key idea is to decouple control learning while preserving a unified autoregressive video prior.<n>For human motion control, we design a t-guided Dynamic Projection mechanism and a refined Motion-CFG strategy.
arXiv Detail & Related papers (2026-06-26T11:08:09Z) - ActWorld: From Explorable to Interactive World Model via Action-Aware Memory [36.88820961480639]
We present ActWorld, an interactive world model that supports mid-rollout object interaction within a chunk-autoregressive framework.<n> Experiments show that ActWorld supports both flexible navigation and rich object interaction within a single model.
arXiv Detail & Related papers (2026-06-16T09:47:32Z) - SymphoMotion: Joint Control of Camera Motion and Object Dynamics for Coherent Video Generation [11.470401259593944]
SymphoMotion is a unified motion-control framework that governs camera trajectories and object dynamics within a single model.<n>We show that SymphoMotion significantly outperforms existing methods in visual fidelity, camera controllability, and object-motion accuracy.
arXiv Detail & Related papers (2026-04-04T12:59:24Z) - WorldCam: Interactive Autoregressive 3D Gaming Worlds with Camera Pose as a Unifying Geometric Representation [47.97929550105451]
We establish camera pose as a unifying geometric representation to ground immediate action control and long-term 3D consistency.<n>Our approach substantially outperforms state-of-the-art interactive gaming world models in action controllability, long-horizon visual quality, and 3D spatial consistency.
arXiv Detail & Related papers (2026-03-17T17:59:56Z) - VerseCrafter: Dynamic Realistic Video World Model with 4D Geometric Control [83.92729346325163]
VerseCrafter is a 4D-aware video world model that enables explicit and coherent control over both camera and object dynamics.<n>Our approach is centered on a novel 4D Geometric Control representation, which encodes the world state through a static background point cloud.<n>These 4D controls are rendered into conditioning signals for a pretrained video diffusion model, enabling the generation of high-fidelity, view-consistent videos.
arXiv Detail & Related papers (2026-01-08T17:28:52Z) - Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation [66.95956271144982]
We present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image.<n>Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames.
arXiv Detail & Related papers (2025-06-04T17:59:04Z) - C-Drag: Chain-of-Thought Driven Motion Controller for Video Generation [81.4106601222722]
Trajectory-based motion control has emerged as an intuitive and efficient approach for controllable video generation.<n>We propose a Chain-of-Thought-based motion controller for controllable video generation, named C-Drag.<n>Our method includes an object perception module and a Chain-of-Thought-based motion reasoning module.
arXiv Detail & Related papers (2025-02-27T08:21:03Z) - Persistent Nature: A Generative Model of Unbounded 3D Worlds [74.51149070418002]
We present an extendable, planar scene layout grid that can be rendered from arbitrary camera poses via a 3D decoder and volume rendering.
Based on this representation, we learn a generative world model solely from single-view internet photos.
Our approach enables scene extrapolation beyond the fixed bounds of current 3D generative models, while also supporting a persistent, camera-independent world representation.
arXiv Detail & Related papers (2023-03-23T17:59:40Z) - Playable Environments: Video Manipulation in Space and Time [98.0621309257937]
We present Playable Environments - a new representation for interactive video generation and manipulation in space and time.
With a single image at inference time, our novel framework allows the user to move objects in 3D while generating a video by providing a sequence of desired actions.
Our method builds an environment state for each frame, which can be manipulated by our proposed action module and decoded back to the image space with volumetric rendering.
arXiv Detail & Related papers (2022-03-03T18:51:05Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.