Learning Spatio-Temporal Downsampling for Effective Video Upscaling
- URL: http://arxiv.org/abs/2203.08140v1
- Date: Tue, 15 Mar 2022 17:59:00 GMT
- Title: Learning Spatio-Temporal Downsampling for Effective Video Upscaling
- Authors: Xiaoyu Xiang, Yapeng Tian, Vijay Rengarajan, Lucas Young, Bo Zhu,
Rakesh Ranjan
- Abstract summary: In this paper, we aim to solve the space-time aliasing problem by learning a-temporal downsampling and upsampling.
Our framework enables a variety of applications, including arbitrary video resampling, blurry frame reconstruction, and efficient video storage.
- Score: 20.07194339353278
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Downsampling is one of the most basic image processing operations. Improper
spatio-temporal downsampling applied on videos can cause aliasing issues such
as moir\'e patterns in space and the wagon-wheel effect in time. Consequently,
the inverse task of upscaling a low-resolution, low frame-rate video in space
and time becomes a challenging ill-posed problem due to information loss and
aliasing artifacts. In this paper, we aim to solve the space-time aliasing
problem by learning a spatio-temporal downsampler. Towards this goal, we
propose a neural network framework that jointly learns spatio-temporal
downsampling and upsampling. It enables the downsampler to retain the key
patterns of the original video and maximizes the reconstruction performance of
the upsampler. To make the downsamping results compatible with popular image
and video storage formats, the downsampling results are encoded to uint8 with a
differentiable quantization layer. To fully utilize the space-time
correspondences, we propose two novel modules for explicit temporal propagation
and space-time feature rearrangement. Experimental results show that our
proposed method significantly boosts the space-time reconstruction quality by
preserving spatial textures and motion patterns in both downsampling and
upscaling. Moreover, our framework enables a variety of applications, including
arbitrary video resampling, blurry frame reconstruction, and efficient video
storage.
Related papers
- You Can Ground Earlier than See: An Effective and Efficient Pipeline for
Temporal Sentence Grounding in Compressed Videos [56.676761067861236]
Given an untrimmed video, temporal sentence grounding aims to locate a target moment semantically according to a sentence query.
Previous respectable works have made decent success, but they only focus on high-level visual features extracted from decoded frames.
We propose a new setting, compressed-domain TSG, which directly utilizes compressed videos rather than fully-decompressed frames as the visual input.
arXiv Detail & Related papers (2023-03-14T12:53:27Z) - Towards Interpretable Video Super-Resolution via Alternating
Optimization [115.85296325037565]
We study a practical space-time video super-resolution (STVSR) problem which aims at generating a high-framerate high-resolution sharp video from a low-framerate blurry video.
We propose an interpretable STVSR framework by leveraging both model-based and learning-based methods.
arXiv Detail & Related papers (2022-07-21T21:34:05Z) - Spatial-Temporal Residual Aggregation for High Resolution Video
Inpainting [14.035620730770528]
Recent learning-based inpainting algorithms have achieved compelling results for completing missing regions after removing undesired objects in videos.
We propose STRA-Net, a novel spatial-temporal residual aggregation framework for high resolution video inpainting.
Both the quantitative and qualitative evaluations show that we can produce more temporal-coherent and visually appealing results than the state-of-the-art methods on inpainting high resolution videos.
arXiv Detail & Related papers (2021-11-05T15:50:31Z) - Efficient Space-time Video Super Resolution using Low-Resolution Flow
and Mask Upsampling [12.856102293479486]
This paper aims to generate High-resolution Slow-motion videos from Low Resolution and Low Frame rate videos.
A simplistic solution is the sequential running of Video Super Resolution and Video Frame models.
Our model is lightweight and performs better than current state-of-the-art models in REDS STSR validation set.
arXiv Detail & Related papers (2021-04-12T19:11:57Z) - ARVo: Learning All-Range Volumetric Correspondence for Video Deblurring [92.40655035360729]
Video deblurring models exploit consecutive frames to remove blurs from camera shakes and object motions.
We propose a novel implicit method to learn spatial correspondence among blurry frames in the feature space.
Our proposed method is evaluated on the widely-adopted DVD dataset, along with a newly collected High-Frame-Rate (1000 fps) dataset for Video Deblurring.
arXiv Detail & Related papers (2021-03-07T04:33:13Z) - Motion-blurred Video Interpolation and Extrapolation [72.3254384191509]
We present a novel framework for deblurring, interpolating and extrapolating sharp frames from a motion-blurred video in an end-to-end manner.
To ensure temporal coherence across predicted frames and address potential temporal ambiguity, we propose a simple, yet effective flow-based rule.
arXiv Detail & Related papers (2021-03-04T12:18:25Z) - Learning Spatial and Spatio-Temporal Pixel Aggregations for Image and
Video Denoising [104.59305271099967]
We present a pixel aggregation network and learn the pixel sampling and averaging strategies for image denoising.
We develop a pixel aggregation network for video denoising to sample pixels across the spatial-temporal space.
Our method is able to solve the misalignment issues caused by large motion in dynamic scenes.
arXiv Detail & Related papers (2021-01-26T13:00:46Z) - Across Scales & Across Dimensions: Temporal Super-Resolution using Deep
Internal Learning [11.658606722158517]
We train a video-specific CNN on examples extracted directly from the low-framerate input video.
Our method exploits the strong recurrence of small space-time patches inside a single video sequence.
The higher spatial resolution of video frames provides strong examples as to how to increase the temporal temporal resolution of that video.
arXiv Detail & Related papers (2020-03-19T15:53:01Z) - Zooming Slow-Mo: Fast and Accurate One-Stage Space-Time Video
Super-Resolution [95.26202278535543]
A simple solution is to split it into two sub-tasks: video frame (VFI) and video super-resolution (VSR)
temporalsynthesis and spatial super-resolution are intra-related in this task.
We propose a one-stage space-time video super-resolution framework, which directly synthesizes an HR slow-motion video from an LFR, LR video.
arXiv Detail & Related papers (2020-02-26T16:59:48Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.