Unsupervised Video Interpolation by Learning Multilayered 2.5D Motion Fields

• URL: http://arxiv.org/abs/2204.09900v1
• Date: Thu, 21 Apr 2022 06:17:05 GMT
• Title: Unsupervised Video Interpolation by Learning Multilayered 2.5D Motion Fields
• Authors: Ziang Cheng, Shihao Jiang, Hongdong Li
• Abstract summary: This paper presents a self-supervised approach to video frame learning that requires only a single video. We parameterize the video motions by solving an ordinary differentiable equation (ODE) defined on a time-varying motion field. This implicit neural representation learns the video as a space-time continuum, allowing frame-time continuum at any temporal resolution.
• Score: 75.81417944207806
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: The problem of video frame interpolation is to increase the temporal resolution of a low frame-rate video, by interpolating novel frames between existing temporally sparse frames. This paper presents a self-supervised approach to video frame interpolation that requires only a single video. We pose the video as a set of layers. Each layer is parameterized by two implicit neural networks -- one for learning a static frame and the other for a time-varying motion field corresponding to video dynamics. Together they represent an occlusion-free subset of the scene with a pseudo-depth channel. To model inter-layer occlusions, all layers are lifted to the 2.5D space so that the frontal layer occludes distant layers. This is done by assigning each layer a depth channel, which we call `pseudo-depth', whose partial order defines the occlusion between layers. The pseudo-depths are converted to visibility values through a fully differentiable SoftMin function so that closer layers are more visible than layers in a distance. On the other hand, we parameterize the video motions by solving an ordinary differentiable equation (ODE) defined on a time-varying neural velocity field that guarantees valid motions. This implicit neural representation learns the video as a space-time continuum, allowing frame interpolation at any temporal resolution. We demonstrate the effectiveness of our method on real-world datasets, where our method achieves comparable performance to state-of-the-arts that require ground truth labels for training.

Related papers

• Hierarchical Masked 3D Diffusion Model for Video Outpainting [20.738731220322176]
  We introduce a masked 3D diffusion model for video outpainting. This allows us to use multiple guide frames to connect the results of multiple video clip inferences. We also introduce a hybrid coarse-to-fine inference pipeline to alleviate the artifact accumulation problem.
  arXiv  Detail & Related papers  (2023-09-05T10:52:21Z)
• WALDO: Future Video Synthesis using Object Layer Decomposition and Parametric Flow Prediction [82.79642869586587]
  WALDO is a novel approach to the prediction of future video frames from past ones. Individual images are decomposed into multiple layers combining object masks and a small set of control points. The layer structure is shared across all frames in each video to build dense inter-frame connections.
  arXiv  Detail & Related papers  (2022-11-25T18:59:46Z)
• TTVFI: Learning Trajectory-Aware Transformer for Video Frame Interpolation [50.49396123016185]
  Video frame (VFI) aims to synthesize an intermediate frame between two consecutive frames. We propose a novel Trajectory-aware Transformer for Video Frame Interpolation (TTVFI) Our method outperforms other state-of-the-art methods in four widely-used VFI benchmarks.
  arXiv  Detail & Related papers  (2022-07-19T03:37:49Z)
• Enhanced Deep Animation Video Interpolation [47.7046169124373]
  Existing learning-based frame algorithms extract consecutive frames from high-speed natural videos to train the model. Compared to natural videos, cartoon videos are usually in a low frame rate. We present AutoFI, a method to automatically render training data for deep animation video.
  arXiv  Detail & Related papers  (2022-06-25T14:00:48Z)
• Efficient Video Segmentation Models with Per-frame Inference [117.97423110566963]
  We focus on improving the temporal consistency without introducing overhead in inference. We propose several techniques to learn from the video sequence, including a temporal consistency loss and online/offline knowledge distillation methods.
  arXiv  Detail & Related papers  (2022-02-24T23:51:36Z)
• Video Frame Interpolation without Temporal Priors [91.04877640089053]
  Video frame aims to synthesize non-exist intermediate frames in a video sequence. The temporal priors of videos, i.e. frames per second (FPS) and frame exposure time, may vary from different camera sensors. We devise a novel optical flow refinement strategy for better synthesizing results.
  arXiv  Detail & Related papers  (2021-12-02T12:13:56Z)
• Layered Neural Atlases for Consistent Video Editing [37.69447642502351]
  We present a method that decomposes, or "unwraps", an input video into a set of layered 2D atlases. For each pixel in the video, our method estimates its corresponding 2D coordinate in each of the atlases. We design our atlases to be interpretable and semantic, which facilitates easy and intuitive editing in the atlas domain.
  arXiv  Detail & Related papers  (2021-09-23T14:58:59Z)
• Learning Joint Spatial-Temporal Transformations for Video Inpainting [58.939131620135235]
  We propose to learn a joint Spatial-Temporal Transformer Network (STTN) for video inpainting. We simultaneously fill missing regions in all input frames by self-attention, and propose to optimize STTN by a spatial-temporal adversarial loss.
  arXiv  Detail & Related papers  (2020-07-20T16:35:48Z)
• Across Scales & Across Dimensions: Temporal Super-Resolution using Deep Internal Learning [11.658606722158517]
  We train a video-specific CNN on examples extracted directly from the low-framerate input video. Our method exploits the strong recurrence of small space-time patches inside a single video sequence. The higher spatial resolution of video frames provides strong examples as to how to increase the temporal temporal resolution of that video.
  arXiv  Detail & Related papers  (2020-03-19T15:53:01Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.