Rolling Shutter Correction with Intermediate Distortion Flow Estimation

• URL: http://arxiv.org/abs/2404.06350v1
• Date: Tue, 9 Apr 2024 14:40:54 GMT
• Title: Rolling Shutter Correction with Intermediate Distortion Flow Estimation
• Authors: Mingdeng Cao, Sidi Yang, Yujiu Yang, Yinqiang Zheng,
• Abstract summary: This paper proposes to correct the rolling shutter (RS) distorted images by estimating the distortion flow from the global shutter (GS) to RS directly. Existing methods usually perform correction using the undistortion flow from the RS to GS. We introduce a new framework that directly estimates the distortion flow and rectifies the RS image with the backward warping operation.
• Score: 55.59359977619609
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: This paper proposes to correct the rolling shutter (RS) distorted images by estimating the distortion flow from the global shutter (GS) to RS directly. Existing methods usually perform correction using the undistortion flow from the RS to GS. They initially predict the flow from consecutive RS frames, subsequently rescaling it as the displacement fields from the RS frame to the underlying GS image using time-dependent scaling factors. Following this, RS-aware forward warping is employed to convert the RS image into its GS counterpart. Nevertheless, this strategy is prone to two shortcomings. First, the undistortion flow estimation is rendered inaccurate by merely linear scaling the flow, due to the complex non-linear motion nature. Second, RS-aware forward warping often results in unavoidable artifacts. To address these limitations, we introduce a new framework that directly estimates the distortion flow and rectifies the RS image with the backward warping operation. More specifically, we first propose a global correlation-based flow attention mechanism to estimate the initial distortion flow and GS feature jointly, which are then refined by the following coarse-to-fine decoder layers. Additionally, a multi-distortion flow prediction strategy is integrated to mitigate the issue of inaccurate flow estimation further. Experimental results validate the effectiveness of the proposed method, which outperforms state-of-the-art approaches on various benchmarks while maintaining high efficiency. The project is available at \url{https://github.com/ljzycmd/DFRSC}.

Related papers

• Deep Linear Array Pushbroom Image Restoration: A Degradation Pipeline and Jitter-Aware Restoration Network [26.86292926584254]
  Linear Array Pushbroom (LAP) imaging technology is widely used in the realm of remote sensing. Traditional methods for restoring LAP images, such as algorithms estimating the point spread function (PSF), exhibit limited performance. We propose a Jitter-Aware Restoration Network (JARNet) to remove the distortion and blur in two stages.
  arXiv  Detail & Related papers  (2024-01-16T07:26:26Z)
• Flow Guidance Deformable Compensation Network for Video Frame Interpolation [33.106776459443275]
  We propose a flow guidance deformable compensation network (FGDCN) to overcome the drawbacks of existing motion-based methods. FGDCN decomposes the frame sampling process into two steps: a flow step and a deformation step. Experimental results show that the proposed algorithm achieves excellent performance on various datasets with fewer parameters.
  arXiv  Detail & Related papers  (2022-11-22T09:35:14Z)
• Rolling Shutter Inversion: Bring Rolling Shutter Images to High Framerate Global Shutter Video [111.08121952640766]
  This paper presents a novel deep-learning based solution to the RS temporal super-resolution problem. By leveraging the multi-view geometry relationship of the RS imaging process, our framework successfully achieves high framerate GS generation. Our method can produce high-quality GS image sequences with rich details, outperforming the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-10-06T16:47:12Z)
• Learning Adaptive Warping for Real-World Rolling Shutter Correction [52.45689075940234]
  This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process.
  arXiv  Detail & Related papers  (2022-04-29T05:13:50Z)
• Neural Global Shutter: Learn to Restore Video from a Rolling Shutter Camera with Global Reset Feature [89.57742172078454]
  Rolling shutter (RS) image sensors suffer from geometric distortion when the camera and object undergo motion during capture. In this paper, we investigate using rolling shutter with a global reset feature (RSGR) to restore clean global shutter (GS) videos. This feature enables us to turn the rectification problem into a deblur-like one, getting rid of inaccurate and costly explicit motion estimation.
  arXiv  Detail & Related papers  (2022-04-03T02:49:28Z)
• Bringing Rolling Shutter Images Alive with Dual Reversed Distortion [75.78003680510193]
  Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time. We develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time.
  arXiv  Detail & Related papers  (2022-03-12T14:57:49Z)
• GMFlow: Learning Optical Flow via Global Matching [124.57850500778277]
  We propose a GMFlow framework for learning optical flow estimation. It consists of three main components: a customized Transformer for feature enhancement, a correlation and softmax layer for global feature matching, and a self-attention layer for flow propagation. Our new framework outperforms 32-iteration RAFT's performance on the challenging Sintel benchmark.
  arXiv  Detail & Related papers  (2021-11-26T18:59:56Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.