STint: Self-supervised Temporal Interpolation for Geospatial Data
- URL: http://arxiv.org/abs/2309.00059v1
- Date: Thu, 31 Aug 2023 18:04:50 GMT
- Title: STint: Self-supervised Temporal Interpolation for Geospatial Data
- Authors: Nidhin Harilal, Bri-Mathias Hodge, Aneesh Subramanian, Claire
Monteleoni
- Abstract summary: Supervised and unsupervised techniques have demonstrated the potential for temporal of video data.
Most prevailing temporal techniques hinge on optical flow, which encodes the motion of pixels between video frames.
In this work, we propose an unsupervised temporal technique, which does not rely on ground truth data or require any motion information like optical flow.
- Score: 0.0
- License: http://creativecommons.org/licenses/by/4.0/
- Abstract: Supervised and unsupervised techniques have demonstrated the potential for
temporal interpolation of video data. Nevertheless, most prevailing temporal
interpolation techniques hinge on optical flow, which encodes the motion of
pixels between video frames. On the other hand, geospatial data exhibits lower
temporal resolution while encompassing a spectrum of movements and deformations
that challenge several assumptions inherent to optical flow. In this work, we
propose an unsupervised temporal interpolation technique, which does not rely
on ground truth data or require any motion information like optical flow, thus
offering a promising alternative for better generalization across geospatial
domains. Specifically, we introduce a self-supervised technique of dual cycle
consistency. Our proposed technique incorporates multiple cycle consistency
losses, which result from interpolating two frames between consecutive input
frames through a series of stages. This dual cycle consistent constraint causes
the model to produce intermediate frames in a self-supervised manner. To the
best of our knowledge, this is the first attempt at unsupervised temporal
interpolation without the explicit use of optical flow. Our experimental
evaluations across diverse geospatial datasets show that STint significantly
outperforms existing state-of-the-art methods for unsupervised temporal
interpolation.
Related papers
- OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation [55.676358801492114]
We propose OCAI, a method that supports robust frame ambiguities by generating intermediate video frames alongside optical flows in between.
Our evaluations demonstrate superior quality and enhanced optical flow accuracy on established benchmarks such as Sintel and KITTI.
arXiv Detail & Related papers (2024-03-26T20:23:48Z) - Motion-Aware Video Frame Interpolation [49.49668436390514]
We introduce a Motion-Aware Video Frame Interpolation (MA-VFI) network, which directly estimates intermediate optical flow from consecutive frames.
It not only extracts global semantic relationships and spatial details from input frames with different receptive fields, but also effectively reduces the required computational cost and complexity.
arXiv Detail & Related papers (2024-02-05T11:00:14Z) - SSTM: Spatiotemporal Recurrent Transformers for Multi-frame Optical Flow
Estimation [0.0]
In optical flow estimates in and near cluded regions, and out-of-boundary regions are two of the current significant limitations of optical flow estimation algorithms.
Recent state-of-the-art optical flow estimation algorithms are two-frame based methods where optical flow is estimated sequentially for each consecutive image pair in a sequence.
We propose a learning-based multi-frame optical flow estimation method that estimates two or more consecutive optical flows in parallel from multi-frame image sequences.
arXiv Detail & Related papers (2023-04-26T23:39:40Z) - Unsupervised Learning Optical Flow in Multi-frame Dynamic Environment
Using Temporal Dynamic Modeling [7.111443975103329]
In this paper, we explore the optical flow estimation from multiple-frame sequences of dynamic scenes.
We use motion priors of the adjacent frames to provide more reliable supervision of the occluded regions.
Experiments on KITTI 2012, KITTI 2015, Sintel Clean, and Sintel Final datasets demonstrate the effectiveness of our methods.
arXiv Detail & Related papers (2023-04-14T14:32:02Z) - Blur Interpolation Transformer for Real-World Motion from Blur [52.10523711510876]
We propose a encoded blur transformer (BiT) to unravel the underlying temporal correlation in blur.
Based on multi-scale residual Swin transformer blocks, we introduce dual-end temporal supervision and temporally symmetric ensembling strategies.
In addition, we design a hybrid camera system to collect the first real-world dataset of one-to-many blur-sharp video pairs.
arXiv Detail & Related papers (2022-11-21T13:10:10Z) - Motion-inductive Self-supervised Object Discovery in Videos [99.35664705038728]
We propose a model for processing consecutive RGB frames, and infer the optical flow between any pair of frames using a layered representation.
We demonstrate superior performance over previous state-of-the-art methods on three public video segmentation datasets.
arXiv Detail & Related papers (2022-10-01T08:38:28Z) - Video Interpolation by Event-driven Anisotropic Adjustment of Optical
Flow [11.914613556594725]
We propose an end-to-end training method A2OF for video frame with event-driven Anisotropic Adjustment of Optical Flows.
Specifically, we use events to generate optical flow distribution masks for the intermediate optical flow, which can model the complicated motion between two frames.
arXiv Detail & Related papers (2022-08-19T02:31:33Z) - Video Shadow Detection via Spatio-Temporal Interpolation Consistency
Training [31.115226660100294]
We propose a framework to feed the unlabeled video frames together with the labeled images into an image shadow detection network training.
We then derive the spatial and temporal consistency constraints accordingly for enhancing generalization in the pixel-wise classification.
In addition, we design a Scale-Aware Network for multi-scale shadow knowledge learning in images.
arXiv Detail & Related papers (2022-06-17T14:29:51Z) - TimeLens: Event-based Video Frame Interpolation [54.28139783383213]
We introduce Time Lens, a novel indicates equal contribution method that leverages the advantages of both synthesis-based and flow-based approaches.
We show an up to 5.21 dB improvement in terms of PSNR over state-of-the-art frame-based and event-based methods.
arXiv Detail & Related papers (2021-06-14T10:33:47Z) - FLAVR: Flow-Agnostic Video Representations for Fast Frame Interpolation [97.99012124785177]
FLAVR is a flexible and efficient architecture that uses 3D space-time convolutions to enable end-to-end learning and inference for video framesupervised.
We demonstrate that FLAVR can serve as a useful self- pretext task for action recognition, optical flow estimation, and motion magnification.
arXiv Detail & Related papers (2020-12-15T18:59:30Z)
This list is automatically generated from the titles and abstracts of the papers in this site.
This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.