Real-Time Video Super-Resolution by Joint Local Inference and Global Parameter Estimation

• URL: http://arxiv.org/abs/2105.02794v1
• Date: Thu, 6 May 2021 16:35:09 GMT
• Title: Real-Time Video Super-Resolution by Joint Local Inference and Global Parameter Estimation
• Authors: Noam Elron, Alex Itskovich, Shahar S. Yuval, Noam Levy
• Abstract summary: We present a novel approach to synthesizing training data by simulating two digital-camera image-capture processes at different scales. Our method produces image-pairs in which both images have properties of natural images. We present an efficient CNN architecture, which enables real-time application of video SR on low-power edge-devices.
• Score: 0.0
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: The state of the art in video super-resolution (SR) are techniques based on deep learning, but they perform poorly on real-world videos (see Figure 1). The reason is that training image-pairs are commonly created by downscaling a high-resolution image to produce a low-resolution counterpart. Deep models are therefore trained to undo downscaling and do not generalize to super-resolving real-world images. Several recent publications present techniques for improving the generalization of learning-based SR, but are all ill-suited for real-time application. We present a novel approach to synthesizing training data by simulating two digital-camera image-capture processes at different scales. Our method produces image-pairs in which both images have properties of natural images. Training an SR model using this data leads to far better generalization to real-world images and videos. In addition, deep video-SR models are characterized by a high operations-per-pixel count, which prohibits their application in real-time. We present an efficient CNN architecture, which enables real-time application of video SR on low-power edge-devices. We split the SR task into two sub-tasks: a control-flow which estimates global properties of the input video and adapts the weights and biases of a processing-CNN that performs the actual processing. Since the process-CNN is tailored to the statistics of the input, its capacity kept low, while retaining effectivity. Also, since video-statistics evolve slowly, the control-flow operates at a much lower rate than the video frame-rate. This reduces the overall computational load by as much as two orders of magnitude. This framework of decoupling the adaptivity of the algorithm from the pixel processing, can be applied in a large family of real-time video enhancement applications, e.g., video denoising, local tone-mapping, stabilization, etc.

Related papers

• EfficientSCI: Densely Connected Network with Space-time Factorization for Large-scale Video Snapshot Compressive Imaging [6.8372546605486555]
  We show that an UHD color video with high compression ratio can be reconstructed from a snapshot 2D measurement using a single end-to-end deep learning model with PSNR above 32 dB. Our method significantly outperforms all previous SOTA algorithms with better real-time performance.
  arXiv  Detail & Related papers  (2023-05-17T07:28:46Z)
• Learning Data-Driven Vector-Quantized Degradation Model for Animation Video Super-Resolution [59.71387128485845]
  We explore the characteristics of animation videos and leverage the rich priors in real-world animation data for a more practical animation VSR model. We propose a multi-scale Vector-Quantized Degradation model for animation video Super-Resolution (VQD-SR) to decompose the local details from global structures. A rich-content Real Animation Low-quality (RAL) video dataset is collected for extracting the priors.
  arXiv  Detail & Related papers  (2023-03-17T08:11:14Z)
• VideoINR: Learning Video Implicit Neural Representation for Continuous Space-Time Super-Resolution [75.79379734567604]
  We show that Video Implicit Neural Representation (VideoINR) can be decoded to videos of arbitrary spatial resolution and frame rate. We show that VideoINR achieves competitive performances with state-of-the-art STVSR methods on common up-sampling scales.
  arXiv  Detail & Related papers  (2022-06-09T17:45:49Z)
• Real-Time Super-Resolution for Real-World Images on Mobile Devices [11.632812550056173]
  Image Super-Resolution (ISR) aims at recovering High-Resolution (HR) images from the corresponding Low-Resolution (LR) counterparts. Recent progress in ISR has been remarkable, but they are way too computationally intensive to be deployed on edge devices. In this work, an approach for real-time ISR on mobile devices is presented, which is able to deal with a wide range of degradations in real-world scenarios.
  arXiv  Detail & Related papers  (2022-06-03T18:44:53Z)
• Self-Supervised Deep Blind Video Super-Resolution [46.410705294831374]
  We propose a self-supervised learning method to solve the blind video SR problem. We generate auxiliary paired data from original LR videos according to the image formation of video SR. Experiments show that our method performs favorably against state-of-the-art ones on benchmarks and real-world videos.
  arXiv  Detail & Related papers  (2022-01-19T05:18:44Z)
• Investigating Tradeoffs in Real-World Video Super-Resolution [90.81396836308085]
  Real-world video super-resolution (VSR) models are often trained with diverse degradations to improve generalizability. To alleviate the first tradeoff, we propose a degradation scheme that reduces up to 40% of training time without sacrificing performance. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences.
  arXiv  Detail & Related papers  (2021-11-24T18:58:21Z)
• Dual-reference Training Data Acquisition and CNN Construction for Image Super-Resolution [33.388234549922025]
  We propose a novel method to capture a large set of realistic LR$sim$HR image pairs using real cameras. Our innovation is to shoot images displayed on an ultra-high quality screen at different resolutions. Experimental results show that training a super-resolution CNN by our LR$sim$HR dataset has superior restoration performance than training it by existing datasets on real world images.
  arXiv  Detail & Related papers  (2021-08-05T03:31:50Z)
• Exploiting Raw Images for Real-Scene Super-Resolution [105.18021110372133]
  We study the problem of real-scene single image super-resolution to bridge the gap between synthetic data and real captured images. We propose a method to generate more realistic training data by mimicking the imaging process of digital cameras. We also develop a two-branch convolutional neural network to exploit the radiance information originally-recorded in raw images.
  arXiv  Detail & Related papers  (2021-02-02T16:10:15Z)
• CNNs for JPEGs: A Study in Computational Cost [49.97673761305336]
  Convolutional neural networks (CNNs) have achieved astonishing advances over the past decade. CNNs are capable of learning robust representations of the data directly from the RGB pixels. Deep learning methods capable of learning directly from the compressed domain have been gaining attention in recent years.
  arXiv  Detail & Related papers  (2020-12-26T15:00:10Z)
• DynaVSR: Dynamic Adaptive Blind Video Super-Resolution [60.154204107453914]
  DynaVSR is a novel meta-learning-based framework for real-world video SR. We train a multi-frame downscaling module with various types of synthetic blur kernels, which is seamlessly combined with a video SR network for input-aware adaptation. Experimental results show that DynaVSR consistently improves the performance of the state-of-the-art video SR models by a large margin.
  arXiv  Detail & Related papers  (2020-11-09T15:07:32Z)
• Robust Single-Image Super-Resolution via CNNs and TV-TV Minimization [7.538482310185135]
  Single-image super-resolution is the process of increasing the resolution of an image, obtaining a high-resolution (HR) image from a low-resolution (LR) one. By leveraging large training datasets, convolutional neural networks (CNNs) currently achieve the state-of-the-art performance in this task. We propose to post-process the CNN outputs with an optimization problem that we call TV-TV minimization, which enforces consistency.
  arXiv  Detail & Related papers  (2020-04-02T07:06:55Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.