PhotonSplat: 3D Scene Reconstruction and Colorization from SPAD Sensors

• URL: http://arxiv.org/abs/2506.21680v1
• Date: Thu, 26 Jun 2025 18:04:28 GMT
• Title: PhotonSplat: 3D Scene Reconstruction and Colorization from SPAD Sensors
• Authors: Sai Sri Teja, Sreevidya Chintalapati, Vinayak Gupta, Mukund Varma T, Haejoon Lee, Aswin Sankaranarayanan, Kaushik Mitra,
• Abstract summary: We introduce PhotonSplat, a framework designed to reconstruct 3D scenes directly from SPAD binary images.<n>Our approach incorporates a novel 3D spatial filtering technique to reduce noise in the renderings.<n>We extend our method to incorporate dynamic scene representations, making it suitable for scenes with moving objects.
• Score: 17.636234096112446
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Advances in 3D reconstruction using neural rendering have enabled high-quality 3D capture. However, they often fail when the input imagery is corrupted by motion blur, due to fast motion of the camera or the objects in the scene. This work advances neural rendering techniques in such scenarios by using single-photon avalanche diode (SPAD) arrays, an emerging sensing technology capable of sensing images at extremely high speeds. However, the use of SPADs presents its own set of unique challenges in the form of binary images, that are driven by stochastic photon arrivals. To address this, we introduce PhotonSplat, a framework designed to reconstruct 3D scenes directly from SPAD binary images, effectively navigating the noise vs. blur trade-off. Our approach incorporates a novel 3D spatial filtering technique to reduce noise in the renderings. The framework also supports both no-reference using generative priors and reference-based colorization from a single blurry image, enabling downstream applications such as segmentation, object detection and appearance editing tasks. Additionally, we extend our method to incorporate dynamic scene representations, making it suitable for scenes with moving objects. We further contribute PhotonScenes, a real-world multi-view dataset captured with the SPAD sensors.

Related papers

• RePose-NeRF: Robust Radiance Fields for Mesh Reconstruction under Noisy Camera Poses [0.5328562411561214]
  We propose a robust framework that reconstructs high-quality, editable 3D meshes directly from multi-view images.<n>Our approach jointly refines camera poses while learning an implicit scene representation that captures fine geometric detail and appearance.
  arXiv  Detail & Related papers  (2025-11-11T18:25:58Z)
• DeblurSplat: SfM-free 3D Gaussian Splatting with Event Camera for Robust Deblurring [50.21760380168387]
  We propose the first Structure-from-Motion (SfM)-free deblurring 3D Gaussian Splatting method via event camera, dubbed DeSplat.<n>We leverage the pretrained capability of the dense stereo module (DUSt3R) to directly obtain accurate initial point clouds from blurred images.
  arXiv  Detail & Related papers  (2025-09-23T11:21:54Z)
• Visibility-Uncertainty-guided 3D Gaussian Inpainting via Scene Conceptional Learning [63.94919846010485]
  3D Gaussian inpainting (3DGI) is challenging in effectively leveraging complementary visual and semantic cues from multiple input views.<n>We propose a method that measures the visibility uncertainties of 3D points across different input views and uses them to guide 3DGI.<n>We build a novel 3DGI framework, VISTA, by integrating VISibility-uncerTainty-guided 3DGI with scene conceptuAl learning.
  arXiv  Detail & Related papers  (2025-04-23T06:21:11Z)
• A Plug-and-Play Algorithm for 3D Video Super-Resolution of Single-Photon LiDAR data [5.378429123269604]
  Single-photon avalanche diodes (SPADs) are advanced sensors capable of detecting individual photons and recording their arrival times with picosecond resolution.<n>We propose a novel computational imaging algorithm to improve the 3D reconstruction of moving scenes from SPAD data.
  arXiv  Detail & Related papers  (2024-12-12T16:33:06Z)
• Gaussian Scenes: Pose-Free Sparse-View Scene Reconstruction using Depth-Enhanced Diffusion Priors [8.440211083233528]
  We introduce a generative approach for pose-free (without camera parameters) reconstruction of 360 scenes from a sparse set of 2D images.<n>We propose an image-to-image generative model designed to inpaint missing details and remove artifacts in novel view renders and depth maps of a 3D scene.
  arXiv  Detail & Related papers  (2024-11-24T19:34:58Z)
• GS-Blur: A 3D Scene-Based Dataset for Realistic Image Deblurring [50.72230109855628]
  We propose GS-Blur, a dataset of synthesized realistic blurry images created using a novel approach. We first reconstruct 3D scenes from multi-view images using 3D Gaussian Splatting (3DGS), then render blurry images by moving the camera view along the randomly generated motion trajectories. By adopting various camera trajectories in reconstructing our GS-Blur, our dataset contains realistic and diverse types of blur, offering a large-scale dataset that generalizes well to real-world blur.
  arXiv  Detail & Related papers  (2024-10-31T06:17:16Z)
• SpikeGS: 3D Gaussian Splatting from Spike Streams with High-Speed Camera Motion [46.23575738669567]
  Novel View Synthesis plays a crucial role by generating new 2D renderings from multi-view images of 3D scenes. High-frame-rate dense 3D reconstruction emerges as a vital technique, enabling detailed and accurate modeling of real-world objects or scenes. Spike cameras, a novel type of neuromorphic sensor, continuously record scenes with an ultra-high temporal resolution.
  arXiv  Detail & Related papers  (2024-07-14T03:19:30Z)
• Director3D: Real-world Camera Trajectory and 3D Scene Generation from Text [61.9973218744157]
  We introduce Director3D, a robust open-world text-to-3D generation framework, designed to generate both real-world 3D scenes and adaptive camera trajectories. Experiments demonstrate that Director3D outperforms existing methods, offering superior performance in real-world 3D generation.
  arXiv  Detail & Related papers  (2024-06-25T14:42:51Z)
• Denoising Diffusion via Image-Based Rendering [54.20828696348574]
  We introduce the first diffusion model able to perform fast, detailed reconstruction and generation of real-world 3D scenes. First, we introduce a new neural scene representation, IB-planes, that can efficiently and accurately represent large 3D scenes. Second, we propose a denoising-diffusion framework to learn a prior over this novel 3D scene representation, using only 2D images.
  arXiv  Detail & Related papers  (2024-02-05T19:00:45Z)
• Deblurring 3D Gaussian Splatting [7.315329140016319]
  We propose a novel real-time deblurring framework, Deblurring 3D Gaussian Splatting, using a small Multi-Layer Perceptron (MLP) While Deblurring 3D Gaussian Splatting can still enjoy real-time rendering, it can reconstruct fine and sharp details from blurry images.
  arXiv  Detail & Related papers  (2024-01-01T18:23:51Z)
• Learning Indoor Inverse Rendering with 3D Spatially-Varying Lighting [149.1673041605155]
  We address the problem of jointly estimating albedo, normals, depth and 3D spatially-varying lighting from a single image. Most existing methods formulate the task as image-to-image translation, ignoring the 3D properties of the scene. We propose a unified, learning-based inverse framework that formulates 3D spatially-varying lighting.
  arXiv  Detail & Related papers  (2021-09-13T15:29:03Z)
• 3D Photography using Context-aware Layered Depth Inpainting [50.66235795163143]
  We propose a method for converting a single RGB-D input image into a 3D photo. A learning-based inpainting model synthesizes new local color-and-depth content into the occluded region. The resulting 3D photos can be efficiently rendered with motion parallax.
  arXiv  Detail & Related papers  (2020-04-09T17:59:06Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.