Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting

• URL: http://arxiv.org/abs/2608.28174v1
• Date: Fri, 28 Aug 2026 10:39:52 GMT
• Title: Manifold4D: Denoising on Point Cloud Rendered Manifolds for Video Re-shooting
• Abstract summary: Video re-shooting re-renders a monocular video of a dynamic scene along a user-specified camera trajectory.<n>The dominant recipe supplies the target geometry explicitly.<n>We propose MANIFOLD4D, which injects the render directly into the initial noise of flow matching.
• Score: 16.163736801043942
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video re-shooting re-renders a monocular video of a dynamic scene along a user-specified camera trajectory, and the dominant recipe supplies the target geometry explicitly: per-frame depth lifts the source video into a 4D point cloud, which is rasterized along the trajectory into a point cloud render. Because the render and the source video are both handed to the network as visual conditions, they compete at every denoising step, leaving the model with a trust dilemma --- how much of the render to believe --- which can degrade trajectory control or visual quality on data outside the training distribution. We argue that a render already pixel-aligned with the target view does not need to be supplied as an explicit conditioning stream at all. We propose MANIFOLD4D, which injects the render directly into the initial noise of flow matching, so that generation no longer departs from standard Gaussian noise but from a new noise manifold carrying geometric information, leaving the source video as the only visual condition. The render is thus used exactly once, and the network is never asked to learn how to read it; in subsequent denoising steps the model can focus on the source video. On our DAVIS-Traj benchmark and on the Vista4D evaluation set, MANIFOLD4D attains the best camera-control accuracy on every metric, lowering rotation error by 25% and 27% and translation error by up to 32% over the strongest baseline, while matching it in video fidelity and leading on real-world novel-view photometric quality. In a user study, our method achieves clear advantages in trajectory following and dynamic consistency. The gap widens as the yaw amplitude grows past the training range, and the model still recovers correct dynamic motion from the source video when the render is deliberately corrupted, confirming that the geometric prior guides generation without overriding it.

Related papers

• Vista4D: Video Reshooting with 4D Point Clouds [60.22650781486302]
  Vista4D is a video reshooting framework that grounds the input video and target cameras in a 4D point cloud.<n>We show improved 4D consistency, camera control, and visual quality compared to state-of-the-art baselines.<n>Our method generalizes to real-world applications such as dynamic scene expansion and 4D scene recomposition.
  arXiv  Detail & Related papers  (2026-04-23T17:57:28Z)
• Reshoot-Anything: A Self-Supervised Model for In-the-Wild Video Reshooting [3.1328424544428852]
  We create a framework capable of leveraging internet-scale monocular videos.<n>Our core contribution is the generation of pseudo multi-view training triplets, consisting of a source video, a geometric anchor, and a target video.<n>At inference, our diffusion transformer utilizes a 4D point-cloud derived anchor to achieve state-of-the-art temporal consistency.
  arXiv  Detail & Related papers  (2026-04-23T15:32:56Z)
• LaVR: Scene Latent Conditioned Generative Video Trajectory Re-Rendering using Large 4D Reconstruction Models [52.656349227001925]
  Given a monocular video, the goal of video re-rendering is to generate views of the scene from a novel camera trajectory.<n>Existing methods face two distinct challenges.<n>We propose to address these challenges by using the implicit geometric knowledge embedded in the latent space of a large 4D reconstruction model.
  arXiv  Detail & Related papers  (2026-01-21T05:46:03Z)
• DriveExplorer: Images-Only Decoupled 4D Reconstruction with Progressive Restoration for Driving View Extrapolation [12.714087160353317]
  This paper presents an effective solution for view extrapolation in autonomous driving scenarios.<n>Recent approaches focus on generating shifted novel view images from given viewpoints using diffusion models.<n>Compared with baselines, our method produces higher-quality images at novel extrapolated viewpoints.
  arXiv  Detail & Related papers  (2025-12-30T04:41:56Z)
• Diff4Splat: Controllable 4D Scene Generation with Latent Dynamic Reconstruction Models [79.06910348413861]
  We introduce Diff4Splat, a feed-forward method that synthesizes controllable and explicit 4D scenes from a single image.<n>Given a single input image, a camera trajectory, and an optional text prompt, Diff4Splat directly predicts a deformable 3D Gaussian field that encodes appearance, geometry, and motion.
  arXiv  Detail & Related papers  (2025-11-01T11:16:25Z)
• SEE4D: Pose-Free 4D Generation via Auto-Regressive Video Inpainting [83.5106058182799]
  We introduce SEE4D, a pose-free, trajectory-to-camera framework for 4D world modeling from casual videos.<n>A view-conditional video in model is trained to learn a robust geometry prior to denoising realistically synthesized images.<n>We validate See4D on cross-view video generation and sparse reconstruction benchmarks.
  arXiv  Detail & Related papers  (2025-10-30T17:59:39Z)
• S^2VG: 3D Stereoscopic and Spatial Video Generation via Denoising Frame Matrix [60.060882467801484]
  We present a pose-free and training-free method that leverages an off-the-shelf monocular video generation model to produce immersive 3D videos.<n>Our approach first warps the generated monocular video into pre-defined camera viewpoints using estimated depth information, then applies a novel textitframe matrix inpainting framework.<n>We validate the efficacy of our proposed method by conducting experiments on videos from various generative models, such as Sora, Lumiere, WALT, and Zeroscope.
  arXiv  Detail & Related papers  (2025-08-11T14:50:03Z)
• Restage4D: Reanimating Deformable 3D Reconstruction from a Single Video [56.781766315691854]
  We introduce textbfRestage4D, a geometry-preserving pipeline for video-conditioned 4D restaging.<n>We validate Restage4D on DAVIS and PointOdyssey, demonstrating improved geometry consistency, motion quality, and 3D tracking performance.
  arXiv  Detail & Related papers  (2025-08-08T21:31:51Z)
• PS4PRO: Pixel-to-pixel Supervision for Photorealistic Rendering and Optimization [3.53658451351123]
  We propose to use video frame as the data augmentation method for neural rendering.<n>PS4PRO is trained on diverse video datasets, implicitly modeling camera movement as well as real-world 3D geometry.<n>Our results indicate that our method improves the reconstruction performance on both static and dynamic scenes.
  arXiv  Detail & Related papers  (2025-05-28T17:35:39Z)
• Deblur4DGS: 4D Gaussian Splatting from Blurry Monocular Video [55.704264233274294]
  We propose Deblur4DGS to reconstruct a high-quality 4D model from blurry monocular video.<n>We transform continuous dynamic representations within an exposure time into the exposure time estimation.<n>Beyond novel-view synthesis, Deblur4DGS can be applied to improve blurry video from multiple perspectives.
  arXiv  Detail & Related papers  (2024-12-09T12:02:11Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.