A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift

• URL: http://arxiv.org/abs/2203.09294v1
• Date: Thu, 17 Mar 2022 12:55:45 GMT
• Title: A Differentiable Two-stage Alignment Scheme for Burst Image Reconstruction with Large Shift
• Authors: Shi Guo, Xi Yang, Jianqi Ma, Gaofeng Ren, Lei Zhang
• Abstract summary: Joint denoising and demosaicking (JDD) for burst images, namely JDD-B, has attracted much attention. One key challenge of JDD-B lies in the robust alignment of image frames. We propose a differentiable two-stage alignment scheme sequentially in patch and pixel level for effective JDD-B.
• Score: 13.454711511086261
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Denoising and demosaicking are two essential steps to reconstruct a clean full-color image from the raw data. Recently, joint denoising and demosaicking (JDD) for burst images, namely JDD-B, has attracted much attention by using multiple raw images captured in a short time to reconstruct a single high-quality image. One key challenge of JDD-B lies in the robust alignment of image frames. State-of-the-art alignment methods in feature domain cannot effectively utilize the temporal information of burst images, where large shifts commonly exist due to camera and object motion. In addition, the higher resolution (e.g., 4K) of modern imaging devices results in larger displacement between frames. To address these challenges, we design a differentiable two-stage alignment scheme sequentially in patch and pixel level for effective JDD-B. The input burst images are firstly aligned in the patch level by using a differentiable progressive block matching method, which can estimate the offset between distant frames with small computational cost. Then we perform implicit pixel-wise alignment in full-resolution feature domain to refine the alignment results. The two stages are jointly trained in an end-to-end manner. Extensive experiments demonstrate the significant improvement of our method over existing JDD-B methods. Codes are available at https://github.com/GuoShi28/2StageAlign.

Related papers

• Multi-Sensor Diffusion-Driven Optical Image Translation for Large-Scale Applications [3.4085512042262374]
  We present a novel method that super-resolves large-scale low spatial resolution images into high-resolution equivalents from disparate optical sensors. Our contributions lie in new forward and reverse diffusion processes, which are crucial for addressing the challenges of large-scale image translation. The outcome is a high-resolution large-scale image with consistent patches, vital for applications such as heterogeneous change detection.
  arXiv  Detail & Related papers  (2024-04-17T10:49:00Z)
• Deep Linear Array Pushbroom Image Restoration: A Degradation Pipeline and Jitter-Aware Restoration Network [26.86292926584254]
  Linear Array Pushbroom (LAP) imaging technology is widely used in the realm of remote sensing. Traditional methods for restoring LAP images, such as algorithms estimating the point spread function (PSF), exhibit limited performance. We propose a Jitter-Aware Restoration Network (JARNet) to remove the distortion and blur in two stages.
  arXiv  Detail & Related papers  (2024-01-16T07:26:26Z)
• Video Frame Interpolation with Many-to-many Splatting and Spatial Selective Refinement [83.60486465697318]
  We propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames. We extend an M2M++ framework by introducing a flexible Spatial Selective Refinement component, which allows for trading computational efficiency for quality and vice versa.
  arXiv  Detail & Related papers  (2023-10-29T09:09:32Z)
• Any-Size-Diffusion: Toward Efficient Text-Driven Synthesis for Any-Size HD Images [56.17404812357676]
  Stable diffusion, a generative model used in text-to-image synthesis, frequently encounters composition problems when generating images of varying sizes. We propose a two-stage pipeline named Any-Size-Diffusion (ASD), designed to efficiently generate well-composed images of any size. We show that ASD can produce well-structured images of arbitrary sizes, cutting down the inference time by 2x compared to the traditional tiled algorithm.
  arXiv  Detail & Related papers  (2023-08-31T09:27:56Z)
• Improving Misaligned Multi-modality Image Fusion with One-stage Progressive Dense Registration [67.23451452670282]
  Misalignments between multi-modality images pose challenges in image fusion. We propose a Cross-modality Multi-scale Progressive Dense Registration scheme. This scheme accomplishes the coarse-to-fine registration exclusively using a one-stage optimization.
  arXiv  Detail & Related papers  (2023-08-22T03:46:24Z)
• Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction [54.00007868515432]
  Existing methods face challenges in estimating the accurate correction field due to the uniform velocity assumption. We propose a geometry-based Quadratic Rolling Shutter (QRS) motion solver, which precisely estimates the high-order correction field of individual pixels. Our method surpasses the state-of-the-art by +4.98, +0.77, and +4.33 of PSNR on Carla-RS, Fastec-RS, and BS-RSC datasets, respectively.
  arXiv  Detail & Related papers  (2023-03-31T15:09:18Z)
• Learning Single Image Defocus Deblurring with Misaligned Training Pairs [80.13320797431487]
  We propose a joint deblurring and reblurring learning framework for single image defocus deblurring. Our framework can be applied to boost defocus deblurring networks in terms of both quantitative metrics and visual quality.
  arXiv  Detail & Related papers  (2022-11-26T07:36:33Z)
• Cross-Camera Deep Colorization [10.254243409261898]
  We propose an end-to-end convolutional neural network to align and fuse images from a color-plus-mono dual-camera system. Our method consistently achieves substantial improvements, i.e., around 10dB PSNR gain.
  arXiv  Detail & Related papers  (2022-08-26T11:02:14Z)
• Bringing Rolling Shutter Images Alive with Dual Reversed Distortion [75.78003680510193]
  Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time. We develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time.
  arXiv  Detail & Related papers  (2022-03-12T14:57:49Z)
• Progressive and Selective Fusion Network for High Dynamic Range Imaging [28.639647755164916]
  This paper considers the problem of generating an HDR image of a scene from its LDR images. Recent studies employ deep learning and solve the problem in an end-to-end fashion, leading to significant performance improvements. We propose a novel method that can better fuse the features based on two ideas. One is multi-step feature fusion; our network gradually fuses the features in a stack of blocks having the same structure.
  arXiv  Detail & Related papers  (2021-08-19T09:42:03Z)
• A Novel Inspection System For Variable Data Printing Using Deep Learning [0.9176056742068814]
  We present a novel approach for inspecting variable data prints (VDP) with an ultra-low false alarm rate (0.005%) The system is based on a comparison between two images: a reference image and an image captured by low-cost scanners.
  arXiv  Detail & Related papers  (2020-01-13T15:07:13Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.