Deep Recurrent Neural Network with Multi-scale Bi-directional Propagation for Video Deblurring

• URL: http://arxiv.org/abs/2112.05150v1
• Date: Thu, 9 Dec 2021 11:02:56 GMT
• Title: Deep Recurrent Neural Network with Multi-scale Bi-directional Propagation for Video Deblurring
• Authors: Chao Zhu, Hang Dong, Jinshan Pan, Boyang Liang, Yuhao Huang, Lean Fu, Fei Wang
• Abstract summary: We propose a deep Recurrent Neural Network with Multi-scale Bi-directional Propagation (RNN-MBP) to propagate and gather information from unaligned neighboring frames for better video deblurring. To better evaluate the proposed algorithm and existing state-of-the-art methods on real-world blurry scenes, we also create a Real-World Blurry Video dataset. The proposed algorithm performs favorably against the state-of-the-art methods on three typical benchmarks.
• Score: 36.94523101375519
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: The success of the state-of-the-art video deblurring methods stems mainly from implicit or explicit estimation of alignment among the adjacent frames for latent video restoration. However, due to the influence of the blur effect, estimating the alignment information from the blurry adjacent frames is not a trivial task. Inaccurate estimations will interfere the following frame restoration. Instead of estimating alignment information, we propose a simple and effective deep Recurrent Neural Network with Multi-scale Bi-directional Propagation (RNN-MBP) to effectively propagate and gather the information from unaligned neighboring frames for better video deblurring. Specifically, we build a Multi-scale Bi-directional Propagation~(MBP) module with two U-Net RNN cells which can directly exploit the inter-frame information from unaligned neighboring hidden states by integrating them in different scales. Moreover, to better evaluate the proposed algorithm and existing state-of-the-art methods on real-world blurry scenes, we also create a Real-World Blurry Video Dataset (RBVD) by a well-designed Digital Video Acquisition System (DVAS) and use it as the training and evaluation dataset. Extensive experimental results demonstrate that the proposed RBVD dataset effectively improves the performance of existing algorithms on real-world blurry videos, and the proposed algorithm performs favorably against the state-of-the-art methods on three typical benchmarks. The code is available at https://github.com/XJTU-CVLAB-LOWLEVEL/RNN-MBP.

Related papers

• DaBiT: Depth and Blur informed Transformer for Joint Refocusing and Super-Resolution [4.332534893042983]
  In many real-world scenarios, recorded videos suffer from accidental focus blur. This paper introduces a framework optimised for focal deblurring (refocusing) and video super-resolution (VSR) We achieve state-of-the-art results with an average PSNR performance over 1.9dB greater than comparable existing video restoration methods.
  arXiv  Detail & Related papers  (2024-07-01T12:22:16Z)
• Efficient Video Deblurring Guided by Motion Magnitude [37.25713728458234]
  We propose a novel framework that utilizes the motion magnitude prior (MMP) as guidance for efficient deep video deblurring. The MMP consists of both spatial and temporal blur level information, which can be further integrated into an efficient recurrent neural network (RNN) for video deblurring.
  arXiv  Detail & Related papers  (2022-07-27T08:57:48Z)
• NSNet: Non-saliency Suppression Sampler for Efficient Video Recognition [89.84188594758588]
  A novel Non-saliency Suppression Network (NSNet) is proposed to suppress the responses of non-salient frames. NSNet achieves the state-of-the-art accuracy-efficiency trade-off and presents a significantly faster (2.44.3x) practical inference speed than state-of-the-art methods.
  arXiv  Detail & Related papers  (2022-07-21T09:41:22Z)
• Optical-Flow-Reuse-Based Bidirectional Recurrent Network for Space-Time Video Super-Resolution [52.899234731501075]
  Space-time video super-resolution (ST-VSR) simultaneously increases the spatial resolution and frame rate for a given video. Existing methods typically suffer from difficulties in how to efficiently leverage information from a large range of neighboring frames. We propose a coarse-to-fine bidirectional recurrent neural network instead of using ConvLSTM to leverage knowledge between adjacent frames.
  arXiv  Detail & Related papers  (2021-10-13T15:21:30Z)
• Efficient Spatio-Temporal Recurrent Neural Network for Video Deblurring [39.63844562890704]
  Real-time deblurring still remains a challenging task due to the complexity of spatially and temporally varying blur itself. We adopt residual dense blocks into RNN cells, so as to efficiently extract the spatial features of the current frame. We contribute a novel dataset (BSD) to the community, by collecting paired/sharp video clips using a co-axis beam splitter acquisition system.
  arXiv  Detail & Related papers  (2021-06-30T12:53:02Z)
• ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring [92.40655035360729]
  Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions. We propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space. Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) dataset for Video Deblurring.
  arXiv  Detail & Related papers  (2021-03-07T04:33:13Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)
• Cascaded Deep Video Deblurring Using Temporal Sharpness Prior [88.98348546566675]
  The proposed algorithm mainly consists of optical flow estimation from intermediate latent frames and latent frame restoration steps. It first develops a deep CNN model to estimate optical flow from intermediate latent frames and then restores the latent frames based on the estimated optical flow. We show that exploring the domain knowledge of video deblurring is able to make the deep CNN model more compact and efficient.
  arXiv  Detail & Related papers  (2020-04-06T09:13:49Z)
• Video Saliency Prediction Using Enhanced Spatiotemporal Alignment Network [35.932447204088845]
  We develop an effective feature alignment network tailored to video saliency prediction (V) The network learns to align the features of the neighboring frames to the reference one in a coarse-to-fine manner. The proposed model is trained end-to-end without any post processing.
  arXiv  Detail & Related papers  (2020-01-02T02:05:35Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.