World Reconstruction From Inconsistent Views

• URL: http://arxiv.org/abs/2603.16736v2
• Date: Wed, 18 Mar 2026 13:16:47 GMT
• Title: World Reconstruction From Inconsistent Views
• Abstract summary: Video diffusion models generate diverse worlds, but individual frames often lack 3D consistency across the output sequence.<n>We propose a new method that handles these inconsistencies by non-rigidly aligning the video frames into a globally-consistent coordinate frame.<n>We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning video models into 3D-consistent world generators.
• Score: 54.12773637505746
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video diffusion models generate high-quality and diverse worlds; however, individual frames often lack 3D consistency across the output sequence, which makes the reconstruction of 3D worlds difficult. To this end, we propose a new method that handles these inconsistencies by non-rigidly aligning the video frames into a globally-consistent coordinate frame that produces sharp and detailed pointcloud reconstructions. First, a geometric foundation model lifts each frame into a pixel-wise 3D pointcloud, which contains unaligned surfaces due to these inconsistencies. We then propose a tailored non-rigid iterative frame-to-model ICP to obtain an initial alignment across all frames, followed by a global optimization that further sharpens the pointcloud. Finally, we leverage this pointcloud as initialization for 3D reconstruction and propose a novel inverse deformation rendering loss to create high quality and explorable 3D environments from inconsistent views. We demonstrate that our 3D scenes achieve higher quality than baselines, effectively turning video models into 3D-consistent world generators.

Related papers

• Axolotl3D: a Unified Framework for Faithful 3D Shape Completion [1.6934548577260198]
  We present Axolotl3D, a 3D generation model that conditions on images, visibility masks, camera parameters, and a partial point cloud.<n>A unified training strategy synthesizes diverse conditioning regimes from large-scale 3D data, enabling robust cross-modal reasoning.<n>Experiments on Toys4K and OmniObject3D demonstrate state-of-the-art performance under both clean and occluded settings.
  arXiv  Detail & Related papers  (2026-07-22T18:38:20Z)
• 3D Consistency Optimization for Self-Supervised Monocular Video Depth Estimation [21.287558280133076]
  We introduce a new paradigm that recasts sequential video depth estimation as an unconstrained multi-view 3D reconstruction problem.<n>Our approach is driven by three constraints: image-level photometric rendering, explicit world-coordinate geometric alignment, and multi-scale temporal gradient consistency.<n>Results show that the proposed approach achieves state-of-the-art spatial accuracy, outperforming the frame-based, video-based depth estimators and the multi-view 3D reconstruction baselines.
  arXiv  Detail & Related papers  (2026-06-14T08:56:45Z)
• Rein3D: Reinforced 3D Indoor Scene Generation with Panoramic Video Diffusion Models [38.45163624089566]
  Rein3D is a framework that reconstructs full 360-degree indoor environments.<n>We employ a radial exploration strategy to render imperfect panoramic videos.<n>These sequences are restored by a panoramic video-to-video diffusion model and further enhanced via video super-resolution to synthesize high-fidelity geometry and textures.
  arXiv  Detail & Related papers  (2026-04-12T10:55:14Z)
• OneWorld: Taming Scene Generation with 3D Unified Representation Autoencoder [90.8453349494245]
  We present OneWorld, a framework that performs diffusion directly within a coherent 3D representation space.<n>OneWorld generates high-quality 3D scenes with superior cross-view consistency compared to state-of-the-art 2D-based methods.
  arXiv  Detail & Related papers  (2026-03-17T03:43:37Z)
• Voyager: Long-Range and World-Consistent Video Diffusion for Explorable 3D Scene Generation [66.95956271144982]
  We present Voyager, a novel video diffusion framework that generates world-consistent 3D point-cloud sequences from a single image.<n>Unlike existing approaches, Voyager achieves end-to-end scene generation and reconstruction with inherent consistency across frames.
  arXiv  Detail & Related papers  (2025-06-04T17:59:04Z)
• Constructing a 3D Scene from a Single Image [31.11317559252235]
  SceneFuse-3D is a training-free framework designed to synthesize coherent 3D scenes from a single top-down view.<n>We decompose the input image into overlapping regions and generate each using a pretrained 3D object generator.<n>This modular design allows us to overcome resolution bottlenecks and preserve spatial structure without requiring 3D supervision or fine-tuning.
  arXiv  Detail & Related papers  (2025-05-21T17:10:47Z)
• LiftImage3D: Lifting Any Single Image to 3D Gaussians with Video Generation Priors [107.83398512719981]
  Single-image 3D reconstruction remains a fundamental challenge in computer vision.<n>Recent advances in Latent Video Diffusion Models offer promising 3D priors learned from large-scale video data.<n>We propose LiftImage3D, a framework that effectively releases LVDMs' generative priors while ensuring 3D consistency.
  arXiv  Detail & Related papers  (2024-12-12T18:58:42Z)
• ReconX: Reconstruct Any Scene from Sparse Views with Video Diffusion Model [16.14713604672497]
  ReconX is a novel 3D scene reconstruction paradigm that reframes the ambiguous reconstruction challenge as a temporal generation task.<n>The proposed ReconX first constructs a global point cloud and encodes it into a contextual space as the 3D structure condition.<n> Guided by the condition, the video diffusion model then synthesizes video frames that are both detail-preserved and exhibit a high degree of 3D consistency.
  arXiv  Detail & Related papers  (2024-08-29T17:59:40Z)
• LAM3D: Large Image-Point-Cloud Alignment Model for 3D Reconstruction from Single Image [64.94932577552458]
  Large Reconstruction Models have made significant strides in the realm of automated 3D content generation from single or multiple input images. Despite their success, these models often produce 3D meshes with geometric inaccuracies, stemming from the inherent challenges of deducing 3D shapes solely from image data. We introduce a novel framework, the Large Image and Point Cloud Alignment Model (LAM3D), which utilizes 3D point cloud data to enhance the fidelity of generated 3D meshes.
  arXiv  Detail & Related papers  (2024-05-24T15:09:12Z)
• 3D-SceneDreamer: Text-Driven 3D-Consistent Scene Generation [51.64796781728106]
  We propose a generative refinement network to synthesize new contents with higher quality by exploiting the natural image prior to 2D diffusion model and the global 3D information of the current scene. Our approach supports wide variety of scene generation and arbitrary camera trajectories with improved visual quality and 3D consistency.
  arXiv  Detail & Related papers  (2024-03-14T14:31:22Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.