ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs

• URL: http://arxiv.org/abs/2506.18792v1
• Date: Mon, 23 Jun 2025 16:01:15 GMT
• Title: ViDAR: Video Diffusion-Aware 4D Reconstruction From Monocular Inputs
• Authors: Michal Nazarczuk, Sibi Catley-Chandar, Thomas Tanay, Zhensong Zhang, Gregory Slabaugh, Eduardo Pérez-Pellitero,
• Abstract summary: We introduce Video-Aware Diffusion Reconstruction (ViDAR), a novel 4D reconstruction framework.<n>ViDAR recovers fine-grained appearance details while mitigating artefacts introduced by monocular ambiguity.<n>Experiments on DyCheck, a challenging benchmark with extreme viewpoint variation, show that ViDAR outperforms all state-of-the-art baselines.
• Score: 7.3647304690955915
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Dynamic Novel View Synthesis aims to generate photorealistic views of moving subjects from arbitrary viewpoints. This task is particularly challenging when relying on monocular video, where disentangling structure from motion is ill-posed and supervision is scarce. We introduce Video Diffusion-Aware Reconstruction (ViDAR), a novel 4D reconstruction framework that leverages personalised diffusion models to synthesise a pseudo multi-view supervision signal for training a Gaussian splatting representation. By conditioning on scene-specific features, ViDAR recovers fine-grained appearance details while mitigating artefacts introduced by monocular ambiguity. To address the spatio-temporal inconsistency of diffusion-based supervision, we propose a diffusion-aware loss function and a camera pose optimisation strategy that aligns synthetic views with the underlying scene geometry. Experiments on DyCheck, a challenging benchmark with extreme viewpoint variation, show that ViDAR outperforms all state-of-the-art baselines in visual quality and geometric consistency. We further highlight ViDAR's strong improvement over baselines on dynamic regions and provide a new benchmark to compare performance in reconstructing motion-rich parts of the scene. Project page: https://vidar-4d.github.io

Related papers

• LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Models [52.656349227001925]
  Given a monocular video, the goal of video re-rendering is to generate views of the scene from a novel camera trajectory.<n>Existing methods face two distinct challenges.<n>We propose to address these challenges by using the implicit geometric knowledge embedded in the latent space of a large 4D reconstruction model.
  arXiv  Detail & Related papers  (2026-01-21T05:46:03Z)
• 4D Driving Scene Generation With Stereo Forcing [62.47705572424127]
  Current generative models struggle to synthesize dynamic 4D driving scenes that simultaneously support temporal extrapolation and spatial novel view synthesis (NVS) without per-scene optimization.<n>We present PhiGenesis, a unified framework for 4D scene generation that extends video generation techniques with geometric and temporal consistency.
  arXiv  Detail & Related papers  (2025-09-24T15:37:17Z)
• SpatialCrafter: Unleashing the Imagination of Video Diffusion Models for Scene Reconstruction from Limited Observations [42.69229582451846]
  This work takes on the challenge of reconstructing 3D scenes from sparse or single-view inputs.<n>We introduce SpatialCrafter, a framework that leverages the rich knowledge in video diffusion models to generate plausible additional observations.<n>Through a trainable camera encoder and an epipolar attention mechanism for explicit geometric constraints, we achieve precise camera control and 3D consistency.
  arXiv  Detail & Related papers  (2025-05-17T13:05:13Z)
• Vivid4D: Improving 4D Reconstruction from Monocular Video by Video Inpainting [26.54811754399946]
  We introduce Vivid4D, a novel approach that enhances 4D monocular video synthesis by augmenting observation views.<n>This reformulates view augmentation as a video inpainting task, where observed views are warped into new viewpoints.<n>Experiments demonstrate that our method effectively improves monocular 4D scene reconstruction and completion.
  arXiv  Detail & Related papers  (2025-04-15T11:38:14Z)
• Decompositional Neural Scene Reconstruction with Generative Diffusion Prior [64.71091831762214]
  Decompositional reconstruction of 3D scenes, with complete shapes and detailed texture, is intriguing for downstream applications.<n>Recent approaches incorporate semantic or geometric regularization to address this issue, but they suffer significant degradation in underconstrained areas.<n>We propose DP-Recon, which employs diffusion priors in the form of Score Distillation Sampling (SDS) to optimize the neural representation of each individual object under novel views.
  arXiv  Detail & Related papers  (2025-03-19T02:11:31Z)
• MultiDiff: Consistent Novel View Synthesis from a Single Image [60.04215655745264]
  MultiDiff is a novel approach for consistent novel view synthesis of scenes from a single RGB image. Our results demonstrate that MultiDiff outperforms state-of-the-art methods on the challenging, real-world datasets RealEstate10K and ScanNet.
  arXiv  Detail & Related papers  (2024-06-26T17:53:51Z)
• DRSM: efficient neural 4d decomposition for dynamic reconstruction in stationary monocular cameras [21.07910546072467]
  We present a novel framework to tackle 4D decomposition problem for dynamic scenes in monocular cameras. Our framework utilizes decomposed static and dynamic feature planes to represent 4D scenes and emphasizes the learning of dynamic regions through dense ray casting.
  arXiv  Detail & Related papers  (2024-02-01T16:38:51Z)
• Diffusion Priors for Dynamic View Synthesis from Monocular Videos [59.42406064983643]
  Dynamic novel view synthesis aims to capture the temporal evolution of visual content within videos. We first finetune a pretrained RGB-D diffusion model on the video frames using a customization technique. We distill the knowledge from the finetuned model to a 4D representations encompassing both dynamic and static Neural Radiance Fields.
  arXiv  Detail & Related papers  (2024-01-10T23:26:41Z)
• MoBluRF: Motion Deblurring Neural Radiance Fields for Blurry Monocular Video [25.964642223641057]
  MoBluRF is a framework for synthesis of sharp-temporal views in blurry monocular video.<n>In the BRI stage, we reconstruct dynamic 3D scenes and jointly initialize the base rays which are used to predict latent sharp rays.<n>In the MDD stage, we introduce a novel Incremental Latent Sharp-rays Prediction (ILSP) approach for the blurry monocular video frames.
  arXiv  Detail & Related papers  (2023-12-21T02:01:19Z)
• Enhanced Stable View Synthesis [86.69338893753886]
  We introduce an approach to enhance the novel view synthesis from images taken from a freely moving camera. The introduced approach focuses on outdoor scenes where recovering accurate geometric scaffold and camera pose is challenging.
  arXiv  Detail & Related papers  (2023-03-30T01:53:14Z)
• Learning Dynamic View Synthesis With Few RGBD Cameras [60.36357774688289]
  We propose to utilize RGBD cameras to synthesize free-viewpoint videos of dynamic indoor scenes. We generate point clouds from RGBD frames and then render them into free-viewpoint videos via a neural feature. We introduce a simple Regional Depth-Inpainting module that adaptively inpaints missing depth values to render complete novel views.
  arXiv  Detail & Related papers  (2022-04-22T03:17:35Z)
• Non-Rigid Neural Radiance Fields: Reconstruction and Novel View Synthesis of a Dynamic Scene From Monocular Video [76.19076002661157]
  Non-Rigid Neural Radiance Fields (NR-NeRF) is a reconstruction and novel view synthesis approach for general non-rigid dynamic scenes. We show that even a single consumer-grade camera is sufficient to synthesize sophisticated renderings of a dynamic scene from novel virtual camera views.
  arXiv  Detail & Related papers  (2020-12-22T18:46:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.