SPC to 3D: Novel View Synthesis from Binary SPC via I2I translation

• URL: http://arxiv.org/abs/2506.06890v1
• Date: Sat, 07 Jun 2025 18:33:21 GMT
• Title: SPC to 3D: Novel View Synthesis from Binary SPC via I2I translation
• Authors: Sumit Sharma, Gopi Raju Matta, Kaushik Mitra,
• Abstract summary: Single Photon Cameras (SPCs) enable image capture at exceptionally high speeds under both low and high illumination.<n>The binary nature of SPC images leads to severe information loss, particularly in texture and color.<n>We propose a modular two-stage framework that converts binary SPC images into high-quality colorized novel views.
• Score: 13.649334929746413
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Single Photon Avalanche Diodes (SPADs) represent a cutting-edge imaging technology, capable of detecting individual photons with remarkable timing precision. Building on this sensitivity, Single Photon Cameras (SPCs) enable image capture at exceptionally high speeds under both low and high illumination. Enabling 3D reconstruction and radiance field recovery from such SPC data holds significant promise. However, the binary nature of SPC images leads to severe information loss, particularly in texture and color, making traditional 3D synthesis techniques ineffective. To address this challenge, we propose a modular two-stage framework that converts binary SPC images into high-quality colorized novel views. The first stage performs image-to-image (I2I) translation using generative models such as Pix2PixHD, converting binary SPC inputs into plausible RGB representations. The second stage employs 3D scene reconstruction techniques like Neural Radiance Fields (NeRF) or Gaussian Splatting (3DGS) to generate novel views. We validate our two-stage pipeline (Pix2PixHD + Nerf/3DGS) through extensive qualitative and quantitative experiments, demonstrating significant improvements in perceptual quality and geometric consistency over the alternative baseline.

Related papers

• EVolSplat: Efficient Volume-based Gaussian Splatting for Urban View Synthesis [61.1662426227688]
  Existing NeRF and 3DGS-based methods show promising results in achieving photorealistic renderings but require slow, per-scene optimization.<n>We introduce EVolSplat, an efficient 3D Gaussian Splatting model for urban scenes that works in a feed-forward manner.
  arXiv  Detail & Related papers  (2025-03-26T02:47:27Z)
• High-Speed Dynamic 3D Imaging with Sensor Fusion Splatting [15.309934457166394]
  Capturing and reconstructing high-speed dynamic 3D scenes has numerous applications in computer graphics, vision, and interdisciplinary fields such as robotics, aerodynamics, and evolutionary biology.<n>Traditional RGB cameras suffer from low frame rates, limited exposure times, and narrow baselines.<n>We propose a novel sensor fusion approach using Gaussian splatting, which combines RGB, depth, and event cameras to capture and reconstruct scenes at high speeds.
  arXiv  Detail & Related papers  (2025-02-07T03:17:31Z)
• Learning Radiance Fields from a Single Snapshot Compressive Image [18.548244681485922]
  Snapshot Compressive Imaging (SCI) technique for recovering the underlying 3D scene structure from a single temporal compressed image.<n>We propose SCINeRF, in which we formulate the physical imaging process of SCI as part of the training of NeRF.<n>We further integrate the popular 3D Gaussian Splatting (3DGS) framework and propose SCISplat to improve 3D scene reconstruction quality and training/rendering speed.
  arXiv  Detail & Related papers  (2024-12-27T06:40:44Z)
• WE-GS: An In-the-wild Efficient 3D Gaussian Representation for Unconstrained Photo Collections [8.261637198675151]
  Novel View Synthesis (NVS) from unconstrained photo collections is challenging in computer graphics. We propose an efficient point-based differentiable rendering framework for scene reconstruction from photo collections. Our approach outperforms existing approaches on the rendering quality of novel view and appearance synthesis with high converge and rendering speed.
  arXiv  Detail & Related papers  (2024-06-04T15:17:37Z)
• Bilateral Guided Radiance Field Processing [4.816861458037213]
  Neural Radiance Fields (NeRF) achieves unprecedented performance in synthesizing novel view synthesis. Image signal processing (ISP) in modern cameras will independently enhance them, leading to "floaters" in the reconstructed radiance fields. We propose to disentangle the enhancement by ISP at the NeRF training stage and re-apply user-desired enhancements to the reconstructed radiance fields. We demonstrate our approach can boost the visual quality of novel view synthesis by effectively removing floaters and performing enhancements from user retouching.
  arXiv  Detail & Related papers  (2024-06-01T14:10:45Z)
• SCINeRF: Neural Radiance Fields from a Snapshot Compressive Image [19.58894449169074]
  Snapshot Compressive Imaging (SCI) technique for recovering the underlying 3D scene representation from a single temporal compressed image. We formulate the physical imaging process of SCI as part of the training of neural radiance fields (NeRF) Our proposed approach surpasses the state-of-the-art methods in terms of image reconstruction and novel view image synthesis.
  arXiv  Detail & Related papers  (2024-03-29T07:14:14Z)
• 2D Gaussian Splatting for Geometrically Accurate Radiance Fields [50.056790168812114]
  3D Gaussian Splatting (3DGS) has recently revolutionized radiance field reconstruction, achieving high quality novel view synthesis and fast rendering speed without baking.<n>We present 2D Gaussian Splatting (2DGS), a novel approach to model and reconstruct geometrically accurate radiance fields from multi-view images.<n>We demonstrate that our differentiable terms allows for noise-free and detailed geometry reconstruction while maintaining competitive appearance quality, fast training speed, and real-time rendering.
  arXiv  Detail & Related papers  (2024-03-26T17:21:24Z)
• StableDreamer: Taming Noisy Score Distillation Sampling for Text-to-3D [88.66678730537777]
  We present StableDreamer, a methodology incorporating three advances. First, we formalize the equivalence of the SDS generative prior and a simple supervised L2 reconstruction loss. Second, our analysis shows that while image-space diffusion contributes to geometric precision, latent-space diffusion is crucial for vivid color rendition.
  arXiv  Detail & Related papers  (2023-12-02T02:27:58Z)
• GS-IR: 3D Gaussian Splatting for Inverse Rendering [71.14234327414086]
  We propose GS-IR, a novel inverse rendering approach based on 3D Gaussian Splatting (GS) We extend GS, a top-performance representation for novel view synthesis, to estimate scene geometry, surface material, and environment illumination from multi-view images captured under unknown lighting conditions. The flexible and expressive GS representation allows us to achieve fast and compact geometry reconstruction, photorealistic novel view synthesis, and effective physically-based rendering.
  arXiv  Detail & Related papers  (2023-11-26T02:35:09Z)
• Real-Time Radiance Fields for Single-Image Portrait View Synthesis [85.32826349697972]
  We present a one-shot method to infer and render a 3D representation from a single unposed image in real-time. Given a single RGB input, our image encoder directly predicts a canonical triplane representation of a neural radiance field for 3D-aware novel view synthesis via volume rendering. Our method is fast (24 fps) on consumer hardware, and produces higher quality results than strong GAN-inversion baselines that require test-time optimization.
  arXiv  Detail & Related papers  (2023-05-03T17:56:01Z)
• Urban Radiance Fields [77.43604458481637]
  We perform 3D reconstruction and novel view synthesis from data captured by scanning platforms commonly deployed for world mapping in urban outdoor environments. Our approach extends Neural Radiance Fields, which has been demonstrated to synthesize realistic novel images for small scenes in controlled settings. Each of these three extensions provides significant performance improvements in experiments on Street View data.
  arXiv  Detail & Related papers  (2021-11-29T15:58:16Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.