Learning Adaptive Warping for Real-World Rolling Shutter Correction
- URL: http://arxiv.org/abs/2204.13886v1
- Date: Fri, 29 Apr 2022 05:13:50 GMT
- Title: Learning Adaptive Warping for Real-World Rolling Shutter Correction
- Authors: Mingdeng Cao, Zhihang Zhong, Jiahao Wang, Yinqiang Zheng, Yujiu Yang
- Abstract summary: This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video.
Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process.
- Score: 52.45689075940234
- License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
- Abstract: This paper proposes the first real-world rolling shutter (RS) correction
dataset, BS-RSC, and a corresponding model to correct the RS frames in a
distorted video. Mobile devices in the consumer market with CMOS-based sensors
for video capture often result in rolling shutter effects when relative
movements occur during the video acquisition process, calling for RS effect
removal techniques. However, current state-of-the-art RS correction methods
often fail to remove RS effects in real scenarios since the motions are various
and hard to model. To address this issue, we propose a real-world RS correction
dataset BS-RSC. Real distorted videos with corresponding ground truth are
recorded simultaneously via a well-designed beam-splitter-based acquisition
system. BS-RSC contains various motions of both camera and objects in dynamic
scenes. Further, an RS correction model with adaptive warping is proposed. Our
model can warp the learned RS features into global shutter counterparts
adaptively with predicted multiple displacement fields. These warped features
are aggregated and then reconstructed into high-quality global shutter frames
in a coarse-to-fine strategy. Experimental results demonstrate the
effectiveness of the proposed method, and our dataset can improve the model's
ability to remove the RS effects in the real world.
Related papers
- SelfDRSC++: Self-Supervised Learning for Dual Reversed Rolling Shutter Correction [72.05587640928879]
We propose an enhanced Self-supervised learning framework for Dual reversed RS distortion Correction (SelfDRSC++)
We introduce a lightweight DRSC network that incorporates a bidirectional correlation matching block to refine the joint optimization of optical flows and corrected RS features.
To effectively train the DRSC network, we propose a self-supervised learning strategy that ensures cycle consistency between input and reconstructed dual reversed RS images.
arXiv Detail & Related papers (2024-08-21T08:17:22Z) - Single Image Rolling Shutter Removal with Diffusion Models [46.57721145372241]
We present RS-Diffusion, the first Diffusion Models-based method for single-frame Rolling Shutter (RS) correction.
In this work, we present an image-to-motion'' framework via diffusion techniques, with a designed patch-attention module.
In addition, we present the RS-Real dataset, comprised of captured RS frames alongside their corresponding Global Shutter (GS) ground-truth pairs.
arXiv Detail & Related papers (2024-07-03T08:25:02Z) - Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and
Events [63.984927609545856]
Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals.
We show that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios.
arXiv Detail & Related papers (2023-04-14T05:30:02Z) - Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter
Correction [54.00007868515432]
Existing methods face challenges in estimating the accurate correction field due to the uniform velocity assumption.
We propose a geometry-based Quadratic Rolling Shutter (QRS) motion solver, which precisely estimates the high-order correction field of individual pixels.
Our method surpasses the state-of-the-art by +4.98, +0.77, and +4.33 of PSNR on Carla-RS, Fastec-RS, and BS-RSC datasets, respectively.
arXiv Detail & Related papers (2023-03-31T15:09:18Z) - Rolling Shutter Inversion: Bring Rolling Shutter Images to High
Framerate Global Shutter Video [111.08121952640766]
This paper presents a novel deep-learning based solution to the RS temporal super-resolution problem.
By leveraging the multi-view geometry relationship of the RS imaging process, our framework successfully achieves high framerate GS generation.
Our method can produce high-quality GS image sequences with rich details, outperforming the state-of-the-art methods.
arXiv Detail & Related papers (2022-10-06T16:47:12Z) - Bringing Rolling Shutter Images Alive with Dual Reversed Distortion [75.78003680510193]
Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time.
We develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time.
arXiv Detail & Related papers (2022-03-12T14:57:49Z) - Deep network for rolling shutter rectification [25.170821013431958]
We propose an end-to-end deep neural network for the challenging task of single image RS rectification.
Our network consists of a motion block, a trajectory module, a row block, an RS rectification module and an RS regeneration module.
Experiments on synthetic and real datasets reveal that our network outperforms prior art both qualitatively and quantitatively.
arXiv Detail & Related papers (2021-12-12T06:40:34Z) - Continuous-Time Spatiotemporal Calibration of a Rolling Shutter
Camera---IMU System [8.201100713224003]
The shutter (RS) mechanism is widely used by consumer-grade cameras, which are essential parts in smartphones and autonomous vehicles.
This work takes the camera-IMU system as an example and looks into the RS effect on itstemporal calibration.
arXiv Detail & Related papers (2021-08-16T16:09:22Z) - Towards Rolling Shutter Correction and Deblurring in Dynamic Scenes [45.663838355778786]
Joint rolling shutter correction and deblurring (RSCD) techniques are critical for CMOS cameras.
We contribute the first dataset, BS-RSCD, which includes both ego-motion and object-motion in dynamic scenes.
Real distorted and blurry videos with corresponding ground truth are recorded simultaneously via a beam-splitter-based acquisition system.
arXiv Detail & Related papers (2021-04-04T12:36:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.