Prediction-assistant Frame Super-Resolution for Video Streaming

• URL: http://arxiv.org/abs/2103.09455v1
• Date: Wed, 17 Mar 2021 06:05:27 GMT
• Title: Prediction-assistant Frame Super-Resolution for Video Streaming
• Authors: Wang Shen, Wenbo Bao, Guangtao Zhai, Charlie L Wang, Jerry W Hu, Zhiyong Gao
• Abstract summary: We propose to enhance video quality using lossy frames in two situations. For the first case, we propose a small yet effective video frame prediction network. For the second case, we improve the video prediction network to associate current frames as well as previous frames to restore high-quality images.
• Score: 40.60863957681011
• License: http://arxiv.org/licenses/nonexclusive-distrib/1.0/
• Abstract: Video frame transmission delay is critical in real-time applications such as online video gaming, live show, etc. The receiving deadline of a new frame must catch up with the frame rendering time. Otherwise, the system will buffer a while, and the user will encounter a frozen screen, resulting in unsatisfactory user experiences. An effective approach is to transmit frames in lower-quality under poor bandwidth conditions, such as using scalable video coding. In this paper, we propose to enhance video quality using lossy frames in two situations. First, when current frames are too late to receive before rendering deadline (i.e., lost), we propose to use previously received high-resolution images to predict the future frames. Second, when the quality of the currently received frames is low~(i.e., lossy), we propose to use previously received high-resolution frames to enhance the low-quality current ones. For the first case, we propose a small yet effective video frame prediction network. For the second case, we improve the video prediction network to a video enhancement network to associate current frames as well as previous frames to restore high-quality images. Extensive experimental results demonstrate that our method performs favorably against state-of-the-art algorithms in the lossy video streaming environment.

Related papers

• ExWarp: Extrapolation and Warping-based Temporal Supersampling for High-frequency Displays [0.7734726150561089]
  High-frequency displays are gaining immense popularity because of their increasing use in video games and virtual reality applications. This paper proposes increasing the frame rate to provide a smooth experience on modern displays by predicting new frames based on past or future frames.
  arXiv  Detail & Related papers  (2023-07-24T08:32:27Z)
• Predictive Coding For Animation-Based Video Compression [13.161311799049978]
  We propose a predictive coding scheme which uses image animation as a predictor, and codes the residual with respect to the actual target frame. Our experiments indicate a significant gain, in excess of 70% compared to the HEVC video standard and over 30% compared to VVC.
  arXiv  Detail & Related papers  (2023-07-09T14:40:54Z)
• Rerender A Video: Zero-Shot Text-Guided Video-to-Video Translation [93.18163456287164]
  This paper proposes a novel text-guided video-to-video translation framework to adapt image models to videos. Our framework achieves global style and local texture temporal consistency at a low cost.
  arXiv  Detail & Related papers  (2023-06-13T17:52:23Z)
• ReBotNet: Fast Real-time Video Enhancement [59.08038313427057]
  Most restoration networks are slow, have high computational bottleneck, and can't be used for real-time video enhancement. In this work, we design an efficient and fast framework to perform real-time enhancement for practical use-cases like live video calls and video streams. To evaluate our method, we emulate two new datasets that real-world video call and streaming scenarios, and show extensive results on multiple datasets where ReBotNet outperforms existing approaches with lower computations, reduced memory requirements, and faster inference time.
  arXiv  Detail & Related papers  (2023-03-23T17:58:05Z)
• FREGAN : an application of generative adversarial networks in enhancing the frame rate of videos [1.1688030627514534]
  FREGAN (Frame Rate Enhancement Generative Adversarial Network) model has been proposed, which predicts future frames of a video sequence based on a sequence of past frames. We have validated the effectiveness of the proposed model on the standard datasets. The experimental outcomes illustrate that the proposed model has a Peak signal-to-noise ratio (PSNR) of 34.94 and a Structural Similarity Index (SSIM) of 0.95.
  arXiv  Detail & Related papers  (2021-11-01T17:19:00Z)
• Memory-Augmented Non-Local Attention for Video Super-Resolution [61.55700315062226]
  We propose a novel video super-resolution method that aims at generating high-fidelity high-resolution (HR) videos from low-resolution (LR) ones. Previous methods predominantly leverage temporal neighbor frames to assist the super-resolution of the current frame. In contrast, we devise a cross-frame non-local attention mechanism that allows video super-resolution without frame alignment.
  arXiv  Detail & Related papers  (2021-08-25T05:12:14Z)
• Motion-blurred Video Interpolation and Extrapolation [72.3254384191509]
  We present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner. To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule.
  arXiv  Detail & Related papers  (2021-03-04T12:18:25Z)
• Is There Tradeoff between Spatial and Temporal in Video Super-Resolution? [50.70768797616581]
  Advanced algorithms have been proposed to exploit the temporal correlation between low-resolution (LR) video frames, and/or to super-resolve a frame with multiple LR frames. These methods pursue higher quality of super-resolved frames, where the quality is usually measured frame by frame in e.g. PSNR. It is a natural requirement to improve both frame-wise fidelity and between-frame consistency, which are termed spatial quality and temporal quality, respectively.
  arXiv  Detail & Related papers  (2020-03-13T07:49:05Z)
• Deep Slow Motion Video Reconstruction with Hybrid Imaging System [12.340049542098148]
  Current techniques increase the frame rate of standard videos through frame by assuming linear object motion which is not valid in challenging cases. We propose a two-stage deep learning system consisting of alignment and appearance estimation. We train our model on synthetically generated hybrid videos and show high-quality results on a variety of test scenes.
  arXiv  Detail & Related papers  (2020-02-27T14:18:12Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.