Single Image Rolling Shutter Removal with Diffusion Models

• URL: http://arxiv.org/abs/2407.02906v1
• Date: Wed, 3 Jul 2024 08:25:02 GMT
• Title: Single Image Rolling Shutter Removal with Diffusion Models
• Authors: Zhanglei Yang, Haipeng Li, Mingbo Hong, Bing Zeng, Shuaicheng Liu,
• Abstract summary: We present RS-Diffusion, the first Diffusion Models-based method for single-frame Rolling Shutter (RS) correction. In this work, we present an image-to-motion'' framework via diffusion techniques, with a designed patch-attention module. In addition, we present the RS-Real dataset, comprised of captured RS frames alongside their corresponding Global Shutter (GS) ground-truth pairs.
• Score: 46.57721145372241
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: We present RS-Diffusion, the first Diffusion Models-based method for single-frame Rolling Shutter (RS) correction. RS artifacts compromise visual quality of frames due to the row wise exposure of CMOS sensors. Most previous methods have focused on multi-frame approaches, using temporal information from consecutive frames for the motion rectification. However, few approaches address the more challenging but important single frame RS correction. In this work, we present an ``image-to-motion'' framework via diffusion techniques, with a designed patch-attention module. In addition, we present the RS-Real dataset, comprised of captured RS frames alongside their corresponding Global Shutter (GS) ground-truth pairs. The GS frames are corrected from the RS ones, guided by the corresponding Inertial Measurement Unit (IMU) gyroscope data acquired during capture. Experiments show that our RS-Diffusion surpasses previous single RS correction methods. Our method and proposed RS-Real dataset lay a solid foundation for advancing the field of RS correction.

Related papers

• Rolling Shutter Correction with Intermediate Distortion Flow Estimation [55.59359977619609]
  This paper proposes to correct the rolling shutter (RS) distorted images by estimating the distortion flow from the global shutter (GS) to RS directly. Existing methods usually perform correction using the undistortion flow from the RS to GS. We introduce a new framework that directly estimates the distortion flow and rectifies the RS image with the backward warping operation.
  arXiv  Detail & Related papers  (2024-04-09T14:40:54Z)
• URS-NeRF: Unordered Rolling Shutter Bundle Adjustment for Neural Radiance Fields [46.186869281594326]
  We propose a novel rolling shutter bundle adjustment method for neural radiance fields (NeRF) We use the unordered rolling shutter (RS) images to obtain the implicit 3D representation.
  arXiv  Detail & Related papers  (2024-03-15T09:08:27Z)
• Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames [6.62974666987451]
  This paper makes the first attempt to tackle the challenging task of recovering arbitrary frame rate latent global shutter (GS) frames from two consecutive rolling shutter (RS) frames. We propose a novel self-supervised framework that leverages events to guide RS frame correction VFI in a unified framework.
  arXiv  Detail & Related papers  (2023-06-27T14:30:25Z)
• Rolling Shutter Inversion: Bring Rolling Shutter Images to High Framerate Global Shutter Video [111.08121952640766]
  This paper presents a novel deep-learning based solution to the RS temporal super-resolution problem. By leveraging the multi-view geometry relationship of the RS imaging process, our framework successfully achieves high framerate GS generation. Our method can produce high-quality GS image sequences with rich details, outperforming the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-10-06T16:47:12Z)
• Learning Adaptive Warping for Real-World Rolling Shutter Correction [52.45689075940234]
  This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process.
  arXiv  Detail & Related papers  (2022-04-29T05:13:50Z)
• Bringing Rolling Shutter Images Alive with Dual Reversed Distortion [75.78003680510193]
  Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time. We develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time.
  arXiv  Detail & Related papers  (2022-03-12T14:57:49Z)
• Deep network for rolling shutter rectification [25.170821013431958]
  We propose an end-to-end deep neural network for the challenging task of single image RS rectification. Our network consists of a motion block, a trajectory module, a row block, an RS rectification module and an RS regeneration module. Experiments on synthetic and real datasets reveal that our network outperforms prior art both qualitatively and quantitatively.
  arXiv  Detail & Related papers  (2021-12-12T06:40:34Z)
• Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes [45.663838355778786]
  Joint rolling shutter correction and deblurring (RSCD) techniques are critical for CMOS cameras. We contribute the first dataset, BS-RSCD, which includes both ego-motion and object-motion in dynamic scenes. Real distorted and blurry videos with corresponding ground truth are recorded simultaneously via a beam-splitter-based acquisition system.
  arXiv  Detail & Related papers  (2021-04-04T12:36:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.