ST-Gen4D: Embedding 4D Spatiotemporal Cognition into World Model for 4D Generation

• URL: http://arxiv.org/abs/2605.07390v1
• Date: Fri, 08 May 2026 07:44:56 GMT
• Title: ST-Gen4D: Embedding 4D Spatiotemporal Cognition into World Model for 4D Generation
• Abstract summary: Generative models have achieved success in producing apparently coherent 2D videos, but remain challenging in the physical world due to lack of 4D scale.<n>Our work proposes ST-4D, a 4D generation framework with 4Dtemporal cognition-based world model.
• Score: 40.29834968806053
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Generative models have achieved success in producing apparently coherent 2D videos, but remain challenging in the physical world due to lack of 4D spatiotemporal scale. Typically, existing 4D generative models directly embed macro scale constraints to enhance overall spatiotemporal consistency. However, these methods only ensure global appearance coherence and fail to reveal the local dynamics of the physical world. Our insight is that global appearance structure and local dynamic topology empower 4D spatiotemporal cognition, thereby enabling 4D generation with spatiotemporal regularities. In this work, we propose ST-Gen4D, a 4D generation framework with 4D spatiotemporal cognition-based world model. Our model is guided by four key designs: 1) Spatiotemporal representation. We encode various modalities into multiple representations as a feature basis. 2) Spatiotemporal cognition. We sculpture these representations into global appearance graph and local dynamic graph, and fuse them via semantic-bridged spatiotemporal fusion to obtain a 4D cognition graph. 3) Spatiotemporal reasoning. We utilize a world model to derive future state based on the 4D cognition. 4) Spatiotemporal generation. We leverage the derived cognition as condition to guide latent diffusion for 4D Gaussian generation. By deeply integrating 4D intrinsic cognition with generative priors, our model guarantees the structural rationality and topological consistency of 4D generation. Moreover, we propose ST-4D datasets by aggregating public 4D datasets and self-built subset. Extensive experiments demonstrate the superiority of our ST-Gen4D across 3D and 4D generation tasks.

Related papers

• Sculpt4D: Generating 4D Shapes via Sparse-Attention Diffusion Transformers [70.55535980817577]
  Sculpt4D is a native 4D generative framework that seamlessly integrates efficient temporal modeling into a pretrained 3D Transformer.<n>We show that Sculpt4D establishes a new state-of-the-art in temporally coherent 4D synthesis and charts a path toward efficient and scalable 4D generation.
  arXiv  Detail & Related papers  (2026-04-23T12:18:55Z)
• Orthogonal Spatial-temporal Distributional Transfer for 4D Generation [58.30004328671699]
  We propose a framework that transfers rich spatial priors from existing 3D diffusion models and temporal priors from video diffusion models to enhance 4D synthesis.<n>We develop a spatial-temporal-disentangled 4D (STD-4D) Diffusion model, which synthesizes 4D-aware videos through disentangled spatial and temporal latents.<n> Experiments demonstrate that our method significantly outperforms existing approaches, achieving superior spatial-temporal consistency and higher-quality 4D synthesis.
  arXiv  Detail & Related papers  (2026-03-05T11:52:21Z)
• SS4D: Native 4D Generative Model via Structured Spacetime Latents [50.29500511908054]
  We present SS4D, a native 4D generative model that synthesizes dynamic 3D objects directly from monocular video.<n>We train a generator directly on 4D data, achieving high fidelity, temporal coherence, and structural consistency.
  arXiv  Detail & Related papers  (2025-12-16T10:45:06Z)
• U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences [54.77163447282599]
  Existing generative frameworks treat all spatial regions uniformly, overlooking the varying uncertainty across real-world scenes.<n>We present U4D, an uncertainty-aware framework for 4D LiDAR world modeling.<n>Our approach first estimates spatial uncertainty maps from a pretrained segmentation model to localize semantically challenging regions.<n>It then performs generation in a "hard-to-easy" manner through two sequential stages: (1) uncertainty-region modeling, which reconstructs high-entropy regions with fine geometric fidelity, and (2) uncertainty-conditioned completion, which synthesizes the remaining areas under learned structural priors.
  arXiv  Detail & Related papers  (2025-12-02T17:59:57Z)
• Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency [49.875459658889355]
  Free4D is a tuning-free framework for 4D scene generation from a single image.<n>Our key insight is to distill pre-trained foundation models for consistent 4D scene representation.<n>The resulting 4D representation enables real-time, controllable rendering.
  arXiv  Detail & Related papers  (2025-03-26T17:59:44Z)
• Advances in 4D Generation: A Survey [23.041037534410773]
  4D generation enables richer interactive and immersive experiences.<n>Despite rapid progress, the field lacks a unified understanding of 4D representations, generative frameworks, basic paradigms, and the core technical challenges it faces.<n>This survey provides a systematic and in-depth review of the 4D generation landscape.
  arXiv  Detail & Related papers  (2025-03-18T17:59:51Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.