Neural Spline Fields for Burst Image Fusion and Layer Separation
- URL: http://arxiv.org/abs/2312.14235v1
- Date: Thu, 21 Dec 2023 18:54:19 GMT
- Title: Neural Spline Fields for Burst Image Fusion and Layer Separation
- Authors: Ilya Chugunov, David Shustin, Ruyu Yan, Chenyang Lei, Felix Heide
- Abstract summary: We propose a versatile intermediate representation: a two-layer alpha-composited image plus flow model constructed with neural spline fields.
Our method is able to jointly fuse a burst image capture into one high-resolution reconstruction and decompose it into transmission and obstruction layers.
We find that, with no post-processing steps or learned priors, our generalizable model is able to outperform existing dedicated single-image and multi-view obstruction removal approaches.
- Score: 40.9442467471977
- License: http://creativecommons.org/publicdomain/zero/1.0/
- Abstract: Each photo in an image burst can be considered a sample of a complex 3D
scene: the product of parallax, diffuse and specular materials, scene motion,
and illuminant variation. While decomposing all of these effects from a stack
of misaligned images is a highly ill-conditioned task, the conventional
align-and-merge burst pipeline takes the other extreme: blending them into a
single image. In this work, we propose a versatile intermediate representation:
a two-layer alpha-composited image plus flow model constructed with neural
spline fields -- networks trained to map input coordinates to spline control
points. Our method is able to, during test-time optimization, jointly fuse a
burst image capture into one high-resolution reconstruction and decompose it
into transmission and obstruction layers. Then, by discarding the obstruction
layer, we can perform a range of tasks including seeing through occlusions,
reflection suppression, and shadow removal. Validated on complex synthetic and
in-the-wild captures we find that, with no post-processing steps or learned
priors, our generalizable model is able to outperform existing dedicated
single-image and multi-view obstruction removal approaches.
Related papers
- Exposure Bracketing is All You Need for Unifying Image Restoration and Enhancement Tasks [50.822601495422916]
We propose to utilize exposure bracketing photography to unify image restoration and enhancement tasks.
Due to the difficulty in collecting real-world pairs, we suggest a solution that first pre-trains the model with synthetic paired data.
In particular, a temporally modulated recurrent network (TMRNet) and self-supervised adaptation method are proposed.
arXiv Detail & Related papers (2024-01-01T14:14:35Z) - Collaborative Blind Image Deblurring [15.555393702795076]
We show that when extracting patches of similar underlying blur is possible, jointly processing the stack of patches yields superior accuracy than handling them separately.
We present three practical patch extraction strategies for image sharpening, camera shake removal and optical aberration correction, and validate the proposed approach on both synthetic and real-world benchmarks.
arXiv Detail & Related papers (2023-05-25T13:14:29Z) - PC-GANs: Progressive Compensation Generative Adversarial Networks for
Pan-sharpening [50.943080184828524]
We propose a novel two-step model for pan-sharpening that sharpens the MS image through the progressive compensation of the spatial and spectral information.
The whole model is composed of triple GANs, and based on the specific architecture, a joint compensation loss function is designed to enable the triple GANs to be trained simultaneously.
arXiv Detail & Related papers (2022-07-29T03:09:21Z) - Single Stage Virtual Try-on via Deformable Attention Flows [51.70606454288168]
Virtual try-on aims to generate a photo-realistic fitting result given an in-shop garment and a reference person image.
We develop a novel Deformable Attention Flow (DAFlow) which applies the deformable attention scheme to multi-flow estimation.
Our proposed method achieves state-of-the-art performance both qualitatively and quantitatively.
arXiv Detail & Related papers (2022-07-19T10:01:31Z) - SVBR-NET: A Non-Blind Spatially Varying Defocus Blur Removal Network [2.4975981795360847]
We propose a non-blind approach for image deblurring that can deal with spatially-varying kernels.
We introduce two encoder-decoder sub-networks that are fed with the blurry image and the estimated blur map.
The network is trained with synthetically blur kernels that are augmented to emulate blur maps produced by existing blur estimation methods.
arXiv Detail & Related papers (2022-06-26T17:21:12Z) - Deep Contrastive Patch-Based Subspace Learning for Camera Image Signal
Processing [5.678834480723395]
We present a specific patch-based, local subspace deep neural network that improves Camera ISP to be robust to heterogeneous artifacts.
We call our three-fold deep-trained model the Patch Subspace Learning Autoencoder (PSL-AE)
PSL-AE encodes patches extracted from noisy a nd clean image pairs, with different artifact types or distortion levels, by contrastive learning.
arXiv Detail & Related papers (2021-04-01T04:40:22Z) - Light Field View Synthesis via Aperture Disparity and Warping Confidence
Map [47.046276641506786]
This paper presents a learning-based approach to synthesize the view from an arbitrary camera position given a sparse set of images.
A key challenge for this novel view synthesis arises from the reconstruction process, when the views from different input images may not be consistent due to obstruction in the light path.
arXiv Detail & Related papers (2020-09-07T09:46:01Z) - Learning to See Through Obstructions with Layered Decomposition [117.77024641706451]
We present a learning-based approach for removing unwanted obstructions from moving images.
Our method leverages motion differences between the background and obstructing elements to recover both layers.
We show that the proposed approach learned from synthetically generated data performs well to real images.
arXiv Detail & Related papers (2020-08-11T17:59:31Z) - Learning to See Through Obstructions [117.77024641706451]
We present a learning-based approach for removing unwanted obstructions from a short sequence of images captured by a moving camera.
Our method leverages the motion differences between the background and the obstructing elements to recover both layers.
We show that training on synthetically generated data transfers well to real images.
arXiv Detail & Related papers (2020-04-02T17:59:12Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.