Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes

• URL: http://arxiv.org/abs/2501.05226v2
• Date: Mon, 13 Jan 2025 15:30:39 GMT
• Title: Light Transport-aware Diffusion Posterior Sampling for Single-View Reconstruction of 3D Volumes
• Authors: Ludwic Leonard, Nils Thuerey, Ruediger Westermann,
• Abstract summary: We introduce a single-view reconstruction technique of fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset. A physically-based differentiable volume is employed to provide volumetric gradients with respect to light transport in the latent space.
• Score: 18.076285588021868
• License:
• Abstract: We introduce a single-view reconstruction technique of volumetric fields in which multiple light scattering effects are omnipresent, such as in clouds. We model the unknown distribution of volumetric fields using an unconditional diffusion model trained on a novel benchmark dataset comprising 1,000 synthetically simulated volumetric density fields. The neural diffusion model is trained on the latent codes of a novel, diffusion-friendly, monoplanar representation. The generative model is used to incorporate a tailored parametric diffusion posterior sampling technique into different reconstruction tasks. A physically-based differentiable volume renderer is employed to provide gradients with respect to light transport in the latent space. This stands in contrast to classic NeRF approaches and makes the reconstructions better aligned with observed data. Through various experiments, we demonstrate single-view reconstruction of volumetric clouds at a previously unattainable quality.

Related papers

• LiftRefine: Progressively Refined View Synthesis from 3D Lifting with Volume-Triplane Representations [21.183524347952762]
  We propose a new view synthesis method via a 3D neural field from both single or few-view input images. Our reconstruction model first lifts one or more input images to the 3D space from a volume as the coarse-scale 3D representation. Our diffusion model then hallucinates missing details in the rendered images from tri-planes.
  arXiv  Detail & Related papers  (2024-12-19T02:23:55Z)
• Steerable Conditional Diffusion for Out-of-Distribution Adaptation in Medical Image Reconstruction [75.91471250967703]
  We introduce a novel sampling framework called Steerable Conditional Diffusion. This framework adapts the diffusion model, concurrently with image reconstruction, based solely on the information provided by the available measurement. We achieve substantial enhancements in out-of-distribution performance across diverse imaging modalities.
  arXiv  Detail & Related papers  (2023-08-28T08:47:06Z)
• Reference-Free Isotropic 3D EM Reconstruction using Diffusion Models [8.590026259176806]
  We propose a diffusion-model-based framework that overcomes the limitations of requiring reference data or prior knowledge about the degradation process. Our approach utilizes 2D diffusion models to consistently reconstruct 3D volumes and is well-suited for highly downsampled data.
  arXiv  Detail & Related papers  (2023-08-03T07:57:02Z)
• Deceptive-NeRF/3DGS: Diffusion-Generated Pseudo-Observations for High-Quality Sparse-View Reconstruction [60.52716381465063]
  We introduce Deceptive-NeRF/3DGS to enhance sparse-view reconstruction with only a limited set of input images. Specifically, we propose a deceptive diffusion model turning noisy images rendered from few-view reconstructions into high-quality pseudo-observations. Our system progressively incorporates diffusion-generated pseudo-observations into the training image sets, ultimately densifying the sparse input observations by 5 to 10 times.
  arXiv  Detail & Related papers  (2023-05-24T14:00:32Z)
• A Variational Perspective on Solving Inverse Problems with Diffusion Models [101.831766524264]
  Inverse tasks can be formulated as inferring a posterior distribution over data. This is however challenging in diffusion models since the nonlinear and iterative nature of the diffusion process renders the posterior intractable. We propose a variational approach that by design seeks to approximate the true posterior distribution.
  arXiv  Detail & Related papers  (2023-05-07T23:00:47Z)
• Single-Stage Diffusion NeRF: A Unified Approach to 3D Generation and Reconstruction [77.69363640021503]
  3D-aware image synthesis encompasses a variety of tasks, such as scene generation and novel view synthesis from images. We present SSDNeRF, a unified approach that employs an expressive diffusion model to learn a generalizable prior of neural radiance fields (NeRF) from multi-view images of diverse objects.
  arXiv  Detail & Related papers  (2023-04-13T17:59:01Z)
• Ensemble flow reconstruction in the atmospheric boundary layer from spatially limited measurements through latent diffusion models [0.32955181898067526]
  Machine learning techniques have previously reconstructed unobserved regions of flow in canonical fluid mechanics problems. These techniques have not yet been demonstrated in the three-dimensional atmospheric boundary layer.
  arXiv  Detail & Related papers  (2023-03-01T21:55:10Z)
• Diffusion Models in Vision: A Survey [73.10116197883303]
  A diffusion model is a deep generative model that is based on two stages, a forward diffusion stage and a reverse diffusion stage. Diffusion models are widely appreciated for the quality and diversity of the generated samples, despite their known computational burdens.
  arXiv  Detail & Related papers  (2022-09-10T22:00:30Z)
• Non-line-of-Sight Imaging via Neural Transient Fields [52.91826472034646]
  We present a neural modeling framework for Non-Line-of-Sight (NLOS) imaging. Inspired by the recent Neural Radiance Field (NeRF) approach, we use a multi-layer perceptron (MLP) to represent the neural transient field or NeTF. We formulate a spherical volume NeTF reconstruction pipeline, applicable to both confocal and non-confocal setups.
  arXiv  Detail & Related papers  (2021-01-02T05:20:54Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.