UniINR: Event-guided Unified Rolling Shutter Correction, Deblurring, and Interpolation

• URL: http://arxiv.org/abs/2305.15078v2
• Date: Thu, 11 Jul 2024 01:31:05 GMT
• Title: UniINR: Event-guided Unified Rolling Shutter Correction, Deblurring, and Interpolation
• Authors: Yunfan LU, Guoqiang Liang, Yusheng Wang, Lin Wang, Hui Xiong,
• Abstract summary: Video frames captured by rolling shutter (RS) cameras during fast camera movement frequently exhibit RS distortion and blur simultaneously. We propose the first and novel approach, named UniINR, to recover arbitrary frame-rate sharp GS frames from an RS blur frame and paired events. Our method features a lightweight model with only 0.38M parameters, and it also enjoys high efficiency, achieving 2.83ms/frame in 31 times frame-rate of an RS blur frame.
• Score: 20.866360240444426
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Video frames captured by rolling shutter (RS) cameras during fast camera movement frequently exhibit RS distortion and blur simultaneously. Naturally, recovering high-frame-rate global shutter (GS) sharp frames from an RS blur frame must simultaneously consider RS correction, deblur, and frame interpolation. A naive way is to decompose the whole process into separate tasks and cascade existing methods; however, this results in cumulative errors and noticeable artifacts. Event cameras enjoy many advantages, e.g., high temporal resolution, making them potential for our problem. To this end, we propose the first and novel approach, named UniINR, to recover arbitrary frame-rate sharp GS frames from an RS blur frame and paired events. Our key idea is unifying spatial-temporal implicit neural representation (INR) to directly map the position and time coordinates to color values to address the interlocking degradations. Specifically, we introduce spatial-temporal implicit encoding (STE) to convert an RS blur image and events into a spatial-temporal representation (STR). To query a specific sharp frame (GS or RS), we embed the exposure time into STR and decode the embedded features pixel-by-pixel to recover a sharp frame. Our method features a lightweight model with only 0.38M parameters, and it also enjoys high inference efficiency, achieving 2.83ms/frame in 31 times frame interpolation of an RS blur frame. Extensive experiments show that our method significantly outperforms prior methods. Code is available at https://github.com/yunfanLu/UniINR.

Related papers

• Video Frame Interpolation with Many-to-many Splatting and Spatial Selective Refinement [83.60486465697318]
  We propose a fully differentiable Many-to-Many (M2M) splatting framework to interpolate frames efficiently. For each input frame pair, M2M has a minuscule computational overhead when interpolating an arbitrary number of in-between frames. We extend an M2M++ framework by introducing a flexible Spatial Selective Refinement component, which allows for trading computational efficiency for quality and vice versa.
  arXiv  Detail & Related papers  (2023-10-29T09:09:32Z)
• An Asynchronous Linear Filter Architecture for Hybrid Event-Frame Cameras [9.69495347826584]
  We present an asynchronous linear filter architecture, fusing event and frame camera data, for HDR video reconstruction and spatial convolution. The proposed AKF pipeline outperforms other state-of-the-art methods in both absolute intensity error (69.4% reduction) and image similarity indexes (average 35.5% improvement)
  arXiv  Detail & Related papers  (2023-09-03T12:37:59Z)
• Self-supervised Learning of Event-guided Video Frame Interpolation for Rolling Shutter Frames [6.62974666987451]
  This paper makes the first attempt to tackle the challenging task of recovering arbitrary frame rate latent global shutter (GS) frames from two consecutive rolling shutter (RS) frames. We propose a novel self-supervised framework that leverages events to guide RS frame correction VFI in a unified framework.
  arXiv  Detail & Related papers  (2023-06-27T14:30:25Z)
• Self-Supervised Scene Dynamic Recovery from Rolling Shutter Images and Events [63.984927609545856]
  Event-based Inter/intra-frame Compensator (E-IC) is proposed to predict the per-pixel dynamic between arbitrary time intervals. We show that the proposed method achieves state-of-the-art and shows remarkable performance for event-based RS2GS inversion in real-world scenarios.
  arXiv  Detail & Related papers  (2023-04-14T05:30:02Z)
• Towards Nonlinear-Motion-Aware and Occlusion-Robust Rolling Shutter Correction [54.00007868515432]
  Existing methods face challenges in estimating the accurate correction field due to the uniform velocity assumption. We propose a geometry-based Quadratic Rolling Shutter (QRS) motion solver, which precisely estimates the high-order correction field of individual pixels. Our method surpasses the state-of-the-art by +4.98, +0.77, and +4.33 of PSNR on Carla-RS, Fastec-RS, and BS-RSC datasets, respectively.
  arXiv  Detail & Related papers  (2023-03-31T15:09:18Z)
• Rolling Shutter Inversion: Bring Rolling Shutter Images to High Framerate Global Shutter Video [111.08121952640766]
  This paper presents a novel deep-learning based solution to the RS temporal super-resolution problem. By leveraging the multi-view geometry relationship of the RS imaging process, our framework successfully achieves high framerate GS generation. Our method can produce high-quality GS image sequences with rich details, outperforming the state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-10-06T16:47:12Z)
• Bringing Rolling Shutter Images Alive with Dual Reversed Distortion [75.78003680510193]
  Rolling shutter (RS) distortion can be interpreted as the result of picking a row of pixels from instant global shutter (GS) frames over time. We develop a novel end-to-end model, IFED, to generate dual optical flow sequence through iterative learning of the velocity field during the RS time.
  arXiv  Detail & Related papers  (2022-03-12T14:57:49Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.