bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction

• URL: http://arxiv.org/abs/2410.23247v3
• Date: Thu, 07 Nov 2024 02:52:47 GMT
• Title: bit2bit: 1-bit quanta video reconstruction via self-supervised photon prediction
• Authors: Yehe Liu, Alexander Krull, Hector Basevi, Ales Leonardis, Michael W. Jenkins,
• Abstract summary: We propose bit2bit, a new method for reconstructing high-quality image stacks at original resolution from sparse binary quantatemporal image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data. We present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions.
• Score: 57.199618102578576
• License:
• Abstract: Quanta image sensors, such as SPAD arrays, are an emerging sensor technology, producing 1-bit arrays representing photon detection events over exposures as short as a few nanoseconds. In practice, raw data are post-processed using heavy spatiotemporal binning to create more useful and interpretable images at the cost of degrading spatiotemporal resolution. In this work, we propose bit2bit, a new method for reconstructing high-quality image stacks at the original spatiotemporal resolution from sparse binary quanta image data. Inspired by recent work on Poisson denoising, we developed an algorithm that creates a dense image sequence from sparse binary photon data by predicting the photon arrival location probability distribution. However, due to the binary nature of the data, we show that the assumption of a Poisson distribution is inadequate. Instead, we model the process with a Bernoulli lattice process from the truncated Poisson. This leads to the proposal of a novel self-supervised solution based on a masked loss function. We evaluate our method using both simulated and real data. On simulated data from a conventional video, we achieve 34.35 mean PSNR with extremely photon-sparse binary input (<0.06 photons per pixel per frame). We also present a novel dataset containing a wide range of real SPAD high-speed videos under various challenging imaging conditions. The scenes cover strong/weak ambient light, strong motion, ultra-fast events, etc., which will be made available to the community, on which we demonstrate the promise of our approach. Both reconstruction quality and throughput substantially surpass the state-of-the-art methods (e.g., Quanta Burst Photography (QBP)). Our approach significantly enhances the visualization and usability of the data, enabling the application of existing analysis techniques.

Related papers

• Transientangelo: Few-Viewpoint Surface Reconstruction Using Single-Photon Lidar [8.464054039931245]
  Lidar captures 3D scene geometry by emitting pulses of light to a target and recording the speed-of-light time delay of the reflected light. conventional lidar systems do not output the raw, captured waveforms of backscattered light. We develop new regularization strategies that improve robustness to photon noise, enabling accurate surface reconstruction with as few as 10 photons per pixel.
  arXiv  Detail & Related papers  (2024-08-22T08:12:09Z)
• Equipping Diffusion Models with Differentiable Spatial Entropy for Low-Light Image Enhancement [7.302792947244082]
  In this work, we propose a novel method that shifts the focus from a deterministic pixel-by-pixel comparison to a statistical perspective. The core idea is to introduce spatial entropy into the loss function to measure the distribution difference between predictions and targets. Specifically, we equip the entropy with diffusion models and aim for superior accuracy and enhanced perceptual quality over l1 based noise matching loss.
  arXiv  Detail & Related papers  (2024-04-15T12:35:10Z)
• Exposure Bracketing is All You Need for Unifying Image Restoration and Enhancement Tasks [50.822601495422916]
  We propose to utilize exposure bracketing photography to unify image restoration and enhancement tasks. Due to the difficulty in collecting real-world pairs, we suggest a solution that first pre-trains the model with synthetic paired data. In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed.
  arXiv  Detail & Related papers  (2024-01-01T14:14:35Z)
• Deep Richardson-Lucy Deconvolution for Low-Light Image Deblurring [48.80983873199214]
  We develop a data-driven approach to model the saturated pixels by a learned latent map. Based on the new model, the non-blind deblurring task can be formulated into a maximum a posterior (MAP) problem. To estimate high-quality deblurred images without amplified artifacts, we develop a prior estimation network.
  arXiv  Detail & Related papers  (2023-08-10T12:53:30Z)
• Curvature regularization for Non-line-of-sight Imaging from Under-sampled Data [5.591221518341613]
  Non-line-of-sight (NLOS) imaging aims to reconstruct the three-dimensional hidden scenes from the data measured in the line-of-sight. We propose novel NLOS reconstruction models based on curvature regularization. We evaluate the proposed algorithms on both synthetic and real datasets.
  arXiv  Detail & Related papers  (2023-01-01T14:10:43Z)
• When the Sun Goes Down: Repairing Photometric Losses for All-Day Depth Estimation [47.617222712429026]
  We show how to use a combination of three techniques to allow the existing photometric losses to work for both day and nighttime images. First, we introduce a per-pixel neural intensity transformation to compensate for the light changes that occur between successive frames. Second, we predict a per-pixel residual flow map that we use to correct the reprojection correspondences induced by the estimated ego-motion and depth.
  arXiv  Detail & Related papers  (2022-06-28T09:29:55Z)
• Deep Unfolded Recovery of Sub-Nyquist Sampled Ultrasound Image [94.42139459221784]
  We propose a reconstruction method from sub-Nyquist samples in the time and spatial domain, that is based on unfolding the ISTA algorithm. Our method allows reducing the number of array elements, sampling rate, and computational time while ensuring high quality imaging performance.
  arXiv  Detail & Related papers  (2021-03-01T19:19:38Z)
• Learning Monocular Dense Depth from Events [53.078665310545745]
  Event cameras produce brightness changes in the form of a stream of asynchronous events instead of intensity frames. Recent learning-based approaches have been applied to event-based data, such as monocular depth prediction. We propose a recurrent architecture to solve this task and show significant improvement over standard feed-forward methods.
  arXiv  Detail & Related papers  (2020-10-16T12:36:23Z)
• CycleISP: Real Image Restoration via Improved Data Synthesis [166.17296369600774]
  We present a framework that models camera imaging pipeline in forward and reverse directions. By training a new image denoising network on realistic synthetic data, we achieve the state-of-the-art performance on real camera benchmark datasets.
  arXiv  Detail & Related papers  (2020-03-17T15:20:25Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.