Neural Global Shutter: Learn to Restore Video from a Rolling Shutter Camera with Global Reset Feature

• URL: http://arxiv.org/abs/2204.00974v1
• Date: Sun, 3 Apr 2022 02:49:28 GMT
• Title: Neural Global Shutter: Learn to Restore Video from a Rolling Shutter Camera with Global Reset Feature
• Authors: Zhixiang Wang, Xiang Ji, Jia-Bin Huang, Shin'ichi Satoh, Xiao Zhou and Yinqiang Zheng
• Abstract summary: Rolling shutter (RS) image sensors suffer from geometric distortion when the camera and object undergo motion during capture. In this paper, we investigate using rolling shutter with a global reset feature (RSGR) to restore clean global shutter (GS) videos. This feature enables us to turn the rectification problem into a deblur-like one, getting rid of inaccurate and costly explicit motion estimation.
• Score: 89.57742172078454
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Most computer vision systems assume distortion-free images as inputs. The widely used rolling-shutter (RS) image sensors, however, suffer from geometric distortion when the camera and object undergo motion during capture. Extensive researches have been conducted on correcting RS distortions. However, most of the existing work relies heavily on the prior assumptions of scenes or motions. Besides, the motion estimation steps are either oversimplified or computationally inefficient due to the heavy flow warping, limiting their applicability. In this paper, we investigate using rolling shutter with a global reset feature (RSGR) to restore clean global shutter (GS) videos. This feature enables us to turn the rectification problem into a deblur-like one, getting rid of inaccurate and costly explicit motion estimation. First, we build an optic system that captures paired RSGR/GS videos. Second, we develop a novel algorithm incorporating spatial and temporal designs to correct the spatial-varying RSGR distortion. Third, we demonstrate that existing image-to-image translation algorithms can recover clean GS videos from distorted RSGR inputs, yet our algorithm achieves the best performance with the specific designs. Our rendered results are not only visually appealing but also beneficial to downstream tasks. Compared to the state-of-the-art RS solution, our RSGR solution is superior in both effectiveness and efficiency. Considering it is easy to realize without changing the hardware, we believe our RSGR solution can potentially replace the RS solution in taking distortion-free videos with low noise and low budget.

Related papers

• Motion-adaptive Separable Collaborative Filters for Blind Motion Deblurring [71.60457491155451]
  Eliminating image blur produced by various kinds of motion has been a challenging problem. We propose a novel real-world deblurring filtering model called the Motion-adaptive Separable Collaborative Filter. Our method provides an effective solution for real-world motion blur removal and achieves state-of-the-art performance.
  arXiv  Detail & Related papers  (2024-04-19T19:44:24Z)
• DeblurGS: Gaussian Splatting for Camera Motion Blur [45.13521168573883]
  We propose DeblurGS, a method to optimize sharp 3D Gaussian Splatting from motion-blurred images. We restore a fine-grained sharp scene by leveraging the remarkable reconstruction capability of 3D Gaussian Splatting. Our approach estimates the 6-Degree-of-Freedom camera motion for each blurry observation and synthesizes corresponding blurry renderings.
  arXiv  Detail & Related papers  (2024-04-17T13:14:52Z)
• Rolling Shutter Correction with Intermediate Distortion Flow Estimation [55.59359977619609]
  This paper proposes to correct the rolling shutter (RS) distorted images by estimating the distortion flow from the global shutter (GS) to RS directly. Existing methods usually perform correction using the undistortion flow from the RS to GS. We introduce a new framework that directly estimates the distortion flow and rectifies the RS image with the backward warping operation.
  arXiv  Detail & Related papers  (2024-04-09T14:40:54Z)
• Toward Efficient Visual Gyroscopes: Spherical Moments, Harmonics Filtering, and Masking Techniques for Spherical Camera Applications [83.8743080143778]
  A visual gyroscope estimates camera rotation through images. The integration of omnidirectional cameras, offering a larger field of view compared to traditional RGB cameras, has proven to yield more accurate and robust results. Here, we address these challenges by introducing a novel visual gyroscope, which combines an Efficient Multi-Mask-Filter Rotation Estor and a Learning based optimization.
  arXiv  Detail & Related papers  (2024-04-02T13:19:06Z)
• Rolling Shutter Inversion: Bring Rolling Shutter Images to High Framerate Global Shutter Video [111.08121952640766]
  This paper presents a novel deep-learning based solution to the RS temporal super-resolution problem. By leveraging the multi-view geometry relationship of the RS imaging process, our framework successfully achieves high framerate GS generation. Our method can produce high-quality GS image sequences with rich details, outperforming the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-10-06T16:47:12Z)
• Revisiting Rolling Shutter Bundle Adjustment: Toward Accurate and Fast Solution [6.317266060165099]
  We propose a robust and fast bundle adjustment solution that estimates the 6-DoF pose of the camera and the geometry of the environment based on measurements from a rolling shutter (RS) camera. This tackles the challenges in the existing works, namely relying on additional sensors, high frame rate video as input, restrictive assumptions on camera motion, readout direction, and poor efficiency.
  arXiv  Detail & Related papers  (2022-09-18T08:21:07Z)
• Learning Adaptive Warping for Real-World Rolling Shutter Correction [52.45689075940234]
  This paper proposes the first real-world rolling shutter (RS) correction dataset, BS-RSC, and a corresponding model to correct the RS frames in a distorted video. Mobile devices in the consumer market with CMOS-based sensors for video capture often result in rolling shutter effects when relative movements occur during the video acquisition process.
  arXiv  Detail & Related papers  (2022-04-29T05:13:50Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.