Continuous Space-Time Video Super-Resolution Utilizing Long-Range Temporal Information

• URL: http://arxiv.org/abs/2302.13256v1
• Date: Sun, 26 Feb 2023 08:02:39 GMT
• Title: Continuous Space-Time Video Super-Resolution Utilizing Long-Range Temporal Information
• Authors: Yuantong Zhang, Daiqin Yang, Zhenzhong Chen, Wenpeng Ding
• Abstract summary: We propose a continuous ST-VSR (CSTVSR) method that can convert the given video to any frame rate and spatial resolution. We show that the proposed algorithm has good flexibility and achieves better performance on various datasets.
• Score: 48.20843501171717
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: In this paper, we consider the task of space-time video super-resolution (ST-VSR), namely, expanding a given source video to a higher frame rate and resolution simultaneously. However, most existing schemes either consider a fixed intermediate time and scale in the training stage or only accept a preset number of input frames (e.g., two adjacent frames) that fails to exploit long-range temporal information. To address these problems, we propose a continuous ST-VSR (C-STVSR) method that can convert the given video to any frame rate and spatial resolution. To achieve time-arbitrary interpolation, we propose a forward warping guided frame synthesis module and an optical-flow-guided context consistency loss to better approximate extreme motion and preserve similar structures among input and prediction frames. In addition, we design a memory-friendly cascading depth-to-space module to realize continuous spatial upsampling. Meanwhile, with the sophisticated reorganization of optical flow, the proposed method is memory friendly, making it possible to propagate information from long-range neighboring frames and achieve better reconstruction quality. Extensive experiments show that the proposed algorithm has good flexibility and achieves better performance on various datasets compared with the state-of-the-art methods in both objective evaluations and subjective visual effects.

Related papers

• CMTA: Cross-Modal Temporal Alignment for Event-guided Video Deblurring [44.30048301161034]
  Video deblurring aims to enhance the quality of restored results in motion-red videos by gathering information from adjacent video frames. We propose two modules: 1) Intra-frame feature enhancement operates within the exposure time of a single blurred frame, and 2) Inter-frame temporal feature alignment gathers valuable long-range temporal information to target frames. We demonstrate that our proposed methods outperform state-of-the-art frame-based and event-based motion deblurring methods through extensive experiments conducted on both synthetic and real-world deblurring datasets.
  arXiv  Detail & Related papers  (2024-08-27T10:09:17Z)
• Video Dehazing via a Multi-Range Temporal Alignment Network with Physical Prior [117.6741444489174]
  Video dehazing aims to recover haze-free frames with high visibility and contrast. This paper presents a novel framework to explore the physical haze priors and aggregate temporal information. We construct the first large-scale outdoor video dehazing benchmark dataset.
  arXiv  Detail & Related papers  (2023-03-17T03:44:17Z)
• Temporal Consistency Learning of inter-frames for Video Super-Resolution [38.26035126565062]
  Video super-resolution (VSR) is a task that aims to reconstruct high-resolution (HR) frames from the low-resolution (LR) reference frame and multiple neighboring frames. Existing methods generally explore information propagation and frame alignment to improve the performance of VSR. We propose a Temporal Consistency learning Network (TCNet) for VSR in an end-to-end manner, to enhance the consistency of the reconstructed videos.
  arXiv  Detail & Related papers  (2022-11-03T08:23:57Z)
• Enhancing Space-time Video Super-resolution via Spatial-temporal Feature Interaction [9.456643513690633]
  The aim of space-time video super-resolution (STVSR) is to increase both the frame rate and the spatial resolution of a video. Recent approaches solve STVSR using end-to-end deep neural networks. We propose a spatial-temporal feature interaction network to enhance STVSR by exploiting both spatial and temporal correlations.
  arXiv  Detail & Related papers  (2022-07-18T22:10:57Z)
• STDAN: Deformable Attention Network for Space-Time Video Super-Resolution [39.18399652834573]
  We propose a deformable attention network called STDAN for STVSR. First, we devise a long-short term feature (LSTFI) module, which is capable of abundant content from more neighboring input frames. Second, we put forward a spatial-temporal deformable feature aggregation (STDFA) module, in which spatial and temporal contexts are adaptively captured and aggregated.
  arXiv  Detail & Related papers  (2022-03-14T03:40:35Z)
• Optical-Flow-Reuse-Based Bidirectional Recurrent Network for Space-Time Video Super-Resolution [52.899234731501075]
  Space-time video super-resolution (ST-VSR) simultaneously increases the spatial resolution and frame rate for a given video. Existing methods typically suffer from difficulties in how to efficiently leverage information from a large range of neighboring frames. We propose a coarse-to-fine bidirectional recurrent neural network instead of using ConvLSTM to leverage knowledge between adjacent frames.
  arXiv  Detail & Related papers  (2021-10-13T15:21:30Z)
• TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
  We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches. We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
  arXiv  Detail & Related papers  (2021-06-14T10:33:47Z)
• FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
  FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised. We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
  arXiv  Detail & Related papers  (2020-12-15T18:59:30Z)
• Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video Super-Resolution [95.26202278535543]
  A simple solution is to split it into two sub-tasks: video frame (VFI) and video super-resolution (VSR) temporalsynthesis and spatial super-resolution are intra-related in this task. We propose a one-stage space-time video super-resolution framework, which directly synthesizes an HR slow-motion video from an LFR, LR video.
  arXiv  Detail & Related papers  (2020-02-26T16:59:48Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.