Beyond Pixels: From Video Priors to 4D Worlds

• URL: http://arxiv.org/abs/2608.10744v1
• Date: Tue, 11 Aug 2026 10:02:41 GMT
• Title: Beyond Pixels: From Video Priors to 4D Worlds
• Abstract summary: 4D generation synthesizes dynamic 3D scenes from conditions such as text or images.<n>Existing methods either generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly.<n>We introduce direct latent-to-4D generation and instantiate it as Latent-to-4D.
• Score: 27.940355511092864
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: 4D generation synthesizes dynamic 3D scenes from conditions such as text or images. Existing methods either reconstruct generated RGB videos with a separate 4D model or adapt a particular video generator to predict geometry directly. The former suffers from distribution mismatch and error propagation, whereas the latter ties 4D prediction to a specific generator and may require retraining when the generator or conditioning regime changes. We ask whether the final denoised latents of video models that share a variational autoencoder (VAE) can instead provide a reusable interface to explicit 4D prediction. Building on this insight, we introduce direct latent-to-4D generation and instantiate it as Latent-to-4D, which bypasses RGB by aligning a video latent with the token grid of a pretrained 4D decoder and refining it through frame-wise and global spatiotemporal attention. Trained on roughly 1K existing reconstruction clips, a single checkpoint transfers unchanged across multiple video diffusion transformers within the same VAE family. On Text4D-200 and I4D-200, Latent-to-4D surpasses matched same-latent Wan+4RC cascades in projection-based DINO-F1 by 2.88--3.45 and 5.81 points, respectively, while also being preferred by human raters for geometry, temporal stability, and overall quality.

Related papers

• Full-4D: Generating Full-Scope 4D Scenes from a Single-View Video [63.61099683012546]
  We introduce RealMV-4D, a dataset of synchronized multi-view videos.<n>We then train a multi-view video diffusion model driven by a novel fused time(T)-view(V) attention mechanism.<n>Experiments demonstrate that our method outperforms existing approaches in both visual fidelity and consistency.
  arXiv  Detail & Related papers  (2026-05-25T06:59:26Z)
• 4RC: 4D Reconstruction via Conditional Querying Anytime and Anywhere [77.83037497484366]
  We present 4RC, a unified feed-forward framework for 4D reconstruction from monocular videos.<n>4RC learns a holistic 4D representation that jointly captures dense scene geometry and motion dynamics.
  arXiv  Detail & Related papers  (2026-02-10T18:57:04Z)
• Any4D: Unified Feed-Forward Metric 4D Reconstruction [39.62006179006032]
  We present Any4D, a scalable multi-view transformer for metric-scale, dense feed-forward 4D reconstruction.<n>Any4D directly generates per-pixel motion and geometry predictions for N frames.<n>We achieve superior performance across diverse setups - both in terms of accuracy (2-3X lower error) and compute efficiency (15X faster)
  arXiv  Detail & Related papers  (2025-12-11T18:57:39Z)
• SWiT-4D: Sliding-Window Transformer for Lossless and Parameter-Free Temporal 4D Generation [30.72482055095692]
  SWiT-4D is a Sliding-Window Transformer for lossless, parameter-free temporal 4D mesh generation.<n> SWiT-4D integrates seamlessly with any Diffusion Transformer (DiT)-based image-to-3D generator.<n>It achieves high-fidelity geometry and stable temporal consistency, indicating practical deployability under extremely limited 4D supervision.
  arXiv  Detail & Related papers  (2025-12-11T17:54:31Z)
• Joint 3D Geometry Reconstruction and Motion Generation for 4D Synthesis from a Single Image [88.71287865590273]
  We introduce TrajScene-60K, a large-scale dataset of 60,000 video samples with dense point trajectories.<n>We propose a diffusion-based 4D Scene Trajectory Generator (4D-STraG) to jointly generate geometrically consistent and motion-plausible 4D trajectories.<n>We then propose a 4D View Synthesis Module (4D-Vi) to render videos with arbitrary camera trajectories from 4D point track representations.
  arXiv  Detail & Related papers  (2025-12-04T17:59:10Z)
• SyncTrack4D: Cross-Video Motion Alignment and Video Synchronization for Multi-Video 4D Gaussian Splatting [50.69165364520998]
  We present a novel multi-video 4D Gaussian Splatting (4DGS) approach designed to handle real-world, unsynchronized video sets.<n>Our approach, SyncTrack4D, directly leverages dense 4D track representation of dynamic scene parts as cues for simultaneous cross-video synchronization and 4DGS reconstruction.<n>We evaluate our approach on the Panoptic Studio and SyncNeRF Blender, demonstrating sub-frame synchronization accuracy with an average temporal error below 0.26 frames, and high-fidelity 4D reconstruction reaching 26.3 PSNR scores.
  arXiv  Detail & Related papers  (2025-12-03T23:05:01Z)
• Dynamic-eDiTor: Training-Free Text-Driven 4D Scene Editing with Multimodal Diffusion Transformer [21.55368174087611]
  We introduce Dynamic-eDiTor, a training-free text-driven 4D editing framework leveraging Multimodal Diffusion Transformer (MM-DiT) and 4DGS.<n>Our method achieves superior editing fidelity and both multi-view and temporal consistency prior approaches.
  arXiv  Detail & Related papers  (2025-11-30T00:18:46Z)
• Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models [79.06910348413861]
  We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image.<n>Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion.
  arXiv  Detail & Related papers  (2025-11-01T11:16:25Z)
• Free4D: Tuning-free 4D Scene Generation with Spatial-Temporal Consistency [49.875459658889355]
  Free4D is a tuning-free framework for 4D scene generation from a single image.<n>Our key insight is to distill pre-trained foundation models for consistent 4D scene representation.<n>The resulting 4D representation enables real-time, controllable rendering.
  arXiv  Detail & Related papers  (2025-03-26T17:59:44Z)
• Consistent4D: Consistent 360{\deg} Dynamic Object Generation from Monocular Video [15.621374353364468]
  Consistent4D is a novel approach for generating 4D dynamic objects from uncalibrated monocular videos. We cast the 360-degree dynamic object reconstruction as a 4D generation problem, eliminating the need for tedious multi-view data collection and camera calibration.
  arXiv  Detail & Related papers  (2023-11-06T03:26:43Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.