SelfDRSC++: Self-Supervised Learning for Dual Reversed Rolling Shutter Correction

• URL: http://arxiv.org/abs/2408.11411v1
• Date: Wed, 21 Aug 2024 08:17:22 GMT
• Title: SelfDRSC++: Self-Supervised Learning for Dual Reversed Rolling Shutter Correction
• Authors: Wei Shang, Dongwei Ren, Wanying Zhang, Qilong Wang, Pengfei Zhu, Wangmeng Zuo,
• Abstract summary: We propose an enhanced Self-supervised learning framework for Dual reversed RS distortion Correction (SelfDRSC++) We introduce a lightweight DRSC network that incorporates a bidirectional correlation matching block to refine the joint optimization of optical flows and corrected RS features. To effectively train the DRSC network, we propose a self-supervised learning strategy that ensures cycle consistency between input and reconstructed dual reversed RS images.
• Score: 72.05587640928879
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Modern consumer cameras commonly employ the rolling shutter (RS) imaging mechanism, via which images are captured by scanning scenes row-by-row, resulting in RS distortion for dynamic scenes. To correct RS distortion, existing methods adopt a fully supervised learning manner that requires high framerate global shutter (GS) images as ground-truth for supervision. In this paper, we propose an enhanced Self-supervised learning framework for Dual reversed RS distortion Correction (SelfDRSC++). Firstly, we introduce a lightweight DRSC network that incorporates a bidirectional correlation matching block to refine the joint optimization of optical flows and corrected RS features, thereby improving correction performance while reducing network parameters. Subsequently, to effectively train the DRSC network, we propose a self-supervised learning strategy that ensures cycle consistency between input and reconstructed dual reversed RS images. The RS reconstruction in SelfDRSC++ can be interestingly formulated as a specialized instance of video frame interpolation, where each row in reconstructed RS images is interpolated from predicted GS images by utilizing RS distortion time maps. By achieving superior performance while simplifying the training process, SelfDRSC++ enables feasible one-stage self-supervised training. Additionally, besides start and end RS scanning time, SelfDRSC++ allows supervision of GS images at arbitrary intermediate scanning times, thus enabling the learned DRSC network to generate high framerate GS videos. The code and trained models are available at \url{https://github.com/shangwei5/SelfDRSC_plusplus}.

Related papers

• Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames [6.62974666987451]
  This paper makes the first attempt to tackle the challenging task of recovering arbitrary frame rate latent global shutter (GS) frames from two consecutive rolling shutter (RS) frames. We propose a novel self-supervised framework that leverages events to guide RS frame correction VFI in a unified framework.
  arXiv  Detail & Related papers  (2023-06-27T14:30:25Z)
• Self-supervised Learning to Bring Dual Reversed Rolling Shutter Images Alive [56.70381414277253]
  We propose a Self-supervised learning framework for Dual RS distortions Correction (SelfDRSC) A DRSC network can be learned to generate a high framerate GS video only based on dual RS images with reversed distortions. On real-world RS cases, our SelfDRSC can produce framerate high-of-the-art videos with finer correction textures and better temporary consistency.
  arXiv  Detail & Related papers  (2023-05-31T13:55:00Z)
• Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events [63.984927609545856]
  Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals. We show that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios.
  arXiv  Detail & Related papers  (2023-04-14T05:30:02Z)
• Rolling Shutter Inversion: Bring Rolling Shutter Images to High Framerate Global Shutter Video [111.08121952640766]
  This paper presents a novel deep-learning based solution to the RS temporal super-resolution problem. By leveraging the multi-view geometry relationship of the RS imaging process, our framework successfully achieves high framerate GS generation. Our method can produce high-quality GS image sequences with rich details, outperforming the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-10-06T16:47:12Z)
• Learning Adaptive Warping for Real-World Rolling Shutter Correction [52.45689075940234]
  This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process.
  arXiv  Detail & Related papers  (2022-04-29T05:13:50Z)
• Bringing Rolling Shutter Images Alive with Dual Reversed Distortion [75.78003680510193]
  Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time. We develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time.
  arXiv  Detail & Related papers  (2022-03-12T14:57:49Z)
• Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes [45.663838355778786]
  Joint rolling shutter correction and deblurring (RSCD) techniques are critical for CMOS cameras. We contribute the first dataset, BS-RSCD, which includes both ego-motion and object-motion in dynamic scenes. Real distorted and blurry videos with corresponding ground truth are recorded simultaneously via a beam-splitter-based acquisition system.
  arXiv  Detail & Related papers  (2021-04-04T12:36:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.