Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in
Time-of-Flight Imaging
- URL: http://arxiv.org/abs/2103.16693v1
- Date: Tue, 30 Mar 2021 21:30:26 GMT
- Title: Mask-ToF: Learning Microlens Masks for Flying Pixel Correction in
Time-of-Flight Imaging
- Authors: Ilya Chugunov, Seung-Hwan Baek, Qiang Fu, Wolfgang Heidrich, Felix
Heide
- Abstract summary: We introduce Mask-ToF, a method to reduce flying pixels (FP) in time-of-flight (ToF) depth captures.
FPs are pervasive artifacts which occur around depth edges, where light paths from both an object and its background are integrated over the aperture.
Mask-ToF learns a microlens-level occlusion mask which effectively creates a custom-shaped sub-aperture for each sensor pixel.
We develop a differentiable ToF simulator to jointly train a convolutional neural network to decode this information and produce high-fidelity, low-FP depth reconstructions.
- Score: 46.09238528698229
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: We introduce Mask-ToF, a method to reduce flying pixels (FP) in
time-of-flight (ToF) depth captures. FPs are pervasive artifacts which occur
around depth edges, where light paths from both an object and its background
are integrated over the aperture. This light mixes at a sensor pixel to produce
erroneous depth estimates, which can adversely affect downstream 3D vision
tasks. Mask-ToF starts at the source of these FPs, learning a microlens-level
occlusion mask which effectively creates a custom-shaped sub-aperture for each
sensor pixel. This modulates the selection of foreground and background light
mixtures on a per-pixel basis and thereby encodes scene geometric information
directly into the ToF measurements. We develop a differentiable ToF simulator
to jointly train a convolutional neural network to decode this information and
produce high-fidelity, low-FP depth reconstructions. We test the effectiveness
of Mask-ToF on a simulated light field dataset and validate the method with an
experimental prototype. To this end, we manufacture the learned amplitude mask
and design an optical relay system to virtually place it on a high-resolution
ToF sensor. We find that Mask-ToF generalizes well to real data without
retraining, cutting FP counts in half.
Related papers
- CodedEvents: Optimal Point-Spread-Function Engineering for 3D-Tracking with Event Cameras [12.329357178025205]
Point-spread-function (PSF) engineering is a well-established computational imaging technique.
We show that existing Fisher phase masks are already near-optimal for localizing static point sources.
We then demonstrate that existing designs are suboptimal for tracking point sources.
arXiv Detail & Related papers (2024-06-13T17:59:46Z) - Improving Lens Flare Removal with General Purpose Pipeline and Multiple
Light Sources Recovery [69.71080926778413]
flare artifacts can affect image visual quality and downstream computer vision tasks.
Current methods do not consider automatic exposure and tone mapping in image signal processing pipeline.
We propose a solution to improve the performance of lens flare removal by revisiting the ISP and design a more reliable light sources recovery strategy.
arXiv Detail & Related papers (2023-08-31T04:58:17Z) - Multi-Modal Neural Radiance Field for Monocular Dense SLAM with a
Light-Weight ToF Sensor [58.305341034419136]
We present the first dense SLAM system with a monocular camera and a light-weight ToF sensor.
We propose a multi-modal implicit scene representation that supports rendering both the signals from the RGB camera and light-weight ToF sensor.
Experiments demonstrate that our system well exploits the signals of light-weight ToF sensors and achieves competitive results.
arXiv Detail & Related papers (2023-08-28T07:56:13Z) - Weakly-Supervised Optical Flow Estimation for Time-of-Flight [11.496094830445054]
We propose a training algorithm, which allows to supervise Optical Flow networks directly on the reconstructed depth.
We demonstrate that this approach enables the training of OF networks to align raw iToF measurements and compensate motion artifacts in the iToF depth images.
arXiv Detail & Related papers (2022-10-11T09:47:23Z) - Progressively-connected Light Field Network for Efficient View Synthesis [69.29043048775802]
We present a Progressively-connected Light Field network (ProLiF) for the novel view synthesis of complex forward-facing scenes.
ProLiF encodes a 4D light field, which allows rendering a large batch of rays in one training step for image- or patch-level losses.
arXiv Detail & Related papers (2022-07-10T13:47:20Z) - Layered Depth Refinement with Mask Guidance [61.10654666344419]
We formulate a novel problem of mask-guided depth refinement that utilizes a generic mask to refine the depth prediction of SIDE models.
Our framework performs layered refinement and inpainting/outpainting, decomposing the depth map into two separate layers signified by the mask and the inverse mask.
We empirically show that our method is robust to different types of masks and initial depth predictions, accurately refining depth values in inner and outer mask boundary regions.
arXiv Detail & Related papers (2022-06-07T06:42:44Z) - End-to-end Learning for Joint Depth and Image Reconstruction from
Diffracted Rotation [10.896567381206715]
We propose a novel end-to-end learning approach for depth from diffracted rotation.
Our approach requires a significantly less complex model and less training data, yet it is superior to existing methods in the task of monocular depth estimation.
arXiv Detail & Related papers (2022-04-14T16:14:37Z) - Facial Depth and Normal Estimation using Single Dual-Pixel Camera [81.02680586859105]
We introduce a DP-oriented Depth/Normal network that reconstructs the 3D facial geometry.
It contains the corresponding ground-truth 3D models including depth map and surface normal in metric scale.
It achieves state-of-the-art performances over recent DP-based depth/normal estimation methods.
arXiv Detail & Related papers (2021-11-25T05:59:27Z) - A Simple Framework for 3D Lensless Imaging with Programmable Masks [37.35255907261072]
We propose a lensless imaging system that captures a small number of measurements using different patterns on a programmable mask.
First, we present a fast recovery algorithm to recover textures on a fixed number of depth planes in the scene.
Second, we consider the mask design problem, for programmable lensless cameras, and provide a design template for optimizing the mask patterns.
Third, we use a refinement network as a post-processing step to identify and remove artifacts in the reconstruction.
arXiv Detail & Related papers (2021-08-18T04:05:33Z) - CodedStereo: Learned Phase Masks for Large Depth-of-field Stereo [24.193656749401075]
Conventional stereo suffers from a fundamental trade-off between imaging volume and signal-to-noise ratio.
We propose a novel end-to-end learning-based technique to overcome this limitation.
We show a 6x increase in volume that can be imaged in simulation.
arXiv Detail & Related papers (2021-04-09T23:44:52Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.