Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation

• URL: http://arxiv.org/abs/2609.03557v1
• Date: Thu, 03 Sep 2026 08:58:04 GMT
• Title: Building Pretraining Data for World Models: An Unreal Engine-Based Pipeline for Action-Conditioned Video Generation
• Abstract summary: We present a large-scale synthetic data production pipeline built on Unreal Engine for generating action-conditioned, multi-view video.<n>The pipeline has produced 2,691 hours of 1080p video and 6,076 hours of 720p video.
• Score: 53.851483224691634
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Action-conditioned video models require large-scale visual data paired with control signals that are temporally aligned with the resulting scene transitions. Such supervision is difficult to obtain from ordinary real-world video because the actions that caused each visual change are typically unknown. We present a large-scale synthetic data production pipeline built on Unreal Engine for generating action-conditioned, multi-view video. To accommodate the different execution requirements of real-time physics and high-quality offline rendering, the pipeline executes trajectory generation and final rendering in two stages: Stage I runs real physics in PIE and records per-frame character states, control inputs, and camera states into an intermediate trajectory representation; Stage II replays those trajectories in a new engine process and renders them offline with Movie Render Queue (MRQ). Around this core, we develop a distributed production system with cache-aware task partitioning, node-local slot scheduling, automated scene screening, aesthetic and luminance filtering, partial-output recovery, asynchronous upload, and continuous cluster health monitoring. The production cluster contains 25 servers with eight NVIDIA RTX 5090 GPUs per server. From 2,384 asset packs, 429 levels were retained for production together with a pool of 40 humanoid characters. The pipeline has produced 2,691 hours of 1080p video and 6,076 hours of 720p video. We describe the system architecture, the implementation decisions that emerged from production failures, and the limitations of using perceptual quality proxies for world-model data curation. The pipeline described in this report constitutes the Unreal Engine synthetic-data production component used in EchoWM.

Related papers

• SolarWM: Open Data and Scalable Training for Long-Horizon Video World Models [88.64689505355055]
  We introduce SolarWM, a fully open foundation for building interactive video world models from data preparation through long-horizon inference.<n>SolarWM addresses this coupling with a reconfigurable multi-source data engine and a backbone-native adaptation framework.<n>We instantiate four 5B--33B models based on Wan2.2, LTX-2.5, and MiniMax-H3 while preserving their native representations and objectives.<n>The resulting causal models enable real-time interaction over rollouts ranging from minutes to hours after being trained on only 5s sequences.
  arXiv  Detail & Related papers  (2026-09-02T17:59:41Z)
• VideoCoCo: Code-as-CoT for Physically-Consistent Video Generation via an Agentic Dual-Engine System [77.32035315337522]
  Text-video models struggle to generate physically consistent dynamics the temporal evolution of a scene must be inferred implicitly from a compressed text prompt.<n>Existing chain-of-thought approaches introduce intermediate plans or visual states, but these representations are typically non-cutable or temporally sparse.<n>We introduce VideoCo, an agentic dual-engine framework in which executable Blender code serves as a process-level chain of thought.
  arXiv  Detail & Related papers  (2026-07-29T18:38:23Z)
• The GEST-Engine: From Event Graphs to Synthetic Video. A Full Technical Report [4.964902130083661]
  We present the GEST-Engine, a complete system that goes from natural-language text to fully-annotated multi-actor video.<n>At its core is an explicit world model: rather than encoding state as a learned latent, the engine maintains a complete, inspectable representation of the world.<n>We describe an in-game world editor, runtime capability extraction, a text-generation pipeline, and a production system that renders corpora at scale across parallel virtual machines.
  arXiv  Detail & Related papers  (2026-07-14T00:25:00Z)
• DriveGen3D: Boosting Feed-Forward Driving Scene Generation with Efficient Video Diffusion [62.589889759543446]
  DriveGen3D is a novel framework for generating high-quality and highly controllable dynamic 3D driving scenes.<n>Our work bridges this methodological gap by integrating accelerated long-term video generation with large-scale dynamic scene reconstruction.
  arXiv  Detail & Related papers  (2025-10-17T03:00:08Z)
• MotionAura: Generating High-Quality and Motion Consistent Videos using Discrete Diffusion [3.7270979204213446]
  We present four key contributions to address the challenges of video processing.<n>First, we introduce the 3D Inverted Vector-Quantization Variencoenco Autocoder.<n>Second, we present MotionAura, a text-to-video generation framework.<n>Third, we propose a spectral transformer-based denoising network.<n>Fourth, we introduce a downstream task of Sketch Guided Videopainting.
  arXiv  Detail & Related papers  (2024-10-10T07:07:56Z)
• xGen-VideoSyn-1: High-fidelity Text-to-Video Synthesis with Compressed Representations [120.52120919834988]
  xGen-SynVideo-1 is a text-to-video (T2V) generation model capable of producing realistic scenes from textual descriptions. VidVAE compresses video data both spatially and temporally, significantly reducing the length of visual tokens. DiT model incorporates spatial and temporal self-attention layers, enabling robust generalization across different timeframes and aspect ratios.
  arXiv  Detail & Related papers  (2024-08-22T17:55:22Z)
• Render In-between: Motion Guided Video Synthesis for Action Interpolation [53.43607872972194]
  We propose a motion-guided frame-upsampling framework that is capable of producing realistic human motion and appearance. A novel motion model is trained to inference the non-linear skeletal motion between frames by leveraging a large-scale motion-capture dataset. Our pipeline only requires low-frame-rate videos and unpaired human motion data but does not require high-frame-rate videos for training.
  arXiv  Detail & Related papers  (2021-11-01T15:32:51Z)
• Video-Data Pipelines for Machine Learning Applications [0.9594432031144714]
  The proposed framework can be scaled to additional video-sequence data sets for ML versioned deployments. We analyze the performance of the proposed video-data pipeline for versioned deployment and monitoring for object detection algorithms.
  arXiv  Detail & Related papers  (2021-10-15T20:28:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.