Optical Flow Estimation in 360$^\circ$ Videos: Dataset, Model and Application

• URL: http://arxiv.org/abs/2301.11880v1
• Date: Fri, 27 Jan 2023 17:50:09 GMT
• Title: Optical Flow Estimation in 360$^\circ$ Videos: Dataset, Model and Application
• Authors: Bin Duan, Keshav Bhandari, Gaowen Liu and Yan Yan
• Abstract summary: We propose the first perceptually realistic 360$circ$ filed-of-view video benchmark dataset, namely FLOW360. We present a novel Siamese representation Learning framework for Omnidirectional Flow (SLOF) estimation, which is trained in a contrastive manner. The learning scheme is further proven to be efficient by expanding our siamese learning scheme and omnidirectional optical flow estimation to the egocentric activity recognition task.
• Score: 9.99133340779672
• License: http://creativecommons.org/licenses/by-nc-nd/4.0/
• Abstract: Optical flow estimation has been a long-lasting and fundamental problem in the computer vision community. However, despite the advances of optical flow estimation in perspective videos, the 360$^\circ$ videos counterpart remains in its infancy, primarily due to the shortage of benchmark datasets and the failure to accommodate the omnidirectional nature of 360$^\circ$ videos. We propose the first perceptually realistic 360$^\circ$ filed-of-view video benchmark dataset, namely FLOW360, with 40 different videos and 4,000 video frames. We then conduct comprehensive characteristic analysis and extensive comparisons with existing datasets, manifesting FLOW360's perceptual realism, uniqueness, and diversity. Moreover, we present a novel Siamese representation Learning framework for Omnidirectional Flow (SLOF) estimation, which is trained in a contrastive manner via a hybrid loss that combines siamese contrastive and optical flow losses. By training the model on random rotations of the input omnidirectional frames, our proposed contrastive scheme accommodates the omnidirectional nature of optical flow estimation in 360$^\circ$ videos, resulting in significantly reduced prediction errors. The learning scheme is further proven to be efficient by expanding our siamese learning scheme and omnidirectional optical flow estimation to the egocentric activity recognition task, where the classification accuracy is boosted up to $\sim$26%. To summarize, we study the optical flow estimation in 360$^\circ$ videos problem from perspectives of the benchmark dataset, learning model, and also practical application. The FLOW360 dataset and code are available at https://siamlof.github.io.

Related papers

• BVI-RLV: A Fully Registered Dataset and Benchmarks for Low-Light Video Enhancement [56.97766265018334]
  This paper introduces a low-light video dataset, consisting of 40 scenes with various motion scenarios under two distinct low-lighting conditions. We provide fully registered ground truth data captured in normal light using a programmable motorized dolly and refine it via an image-based approach for pixel-wise frame alignment across different light levels. Our experimental results demonstrate the significance of fully registered video pairs for low-light video enhancement (LLVE) and the comprehensive evaluation shows that the models trained with our dataset outperform those trained with the existing datasets.
  arXiv  Detail & Related papers  (2024-07-03T22:41:49Z)
• OCAI: Improving Optical Flow Estimation by Occlusion and Consistency Aware Interpolation [55.676358801492114]
  We propose OCAI, a method that supports robust frame ambiguities by generating intermediate video frames alongside optical flows in between. Our evaluations demonstrate superior quality and enhanced optical flow accuracy on established benchmarks such as Sintel and KITTI.
  arXiv  Detail & Related papers  (2024-03-26T20:23:48Z)
• Skin the sheep not only once: Reusing Various Depth Datasets to Drive the Learning of Optical Flow [25.23550076996421]
  We propose to leverage the geometric connection between optical flow estimation and stereo matching. We turn the monocular depth datasets into stereo ones via virtual disparity. We also introduce virtual camera motion into stereo data to produce additional flows along the vertical direction.
  arXiv  Detail & Related papers  (2023-10-03T06:56:07Z)
• Spherical Vision Transformer for 360-degree Video Saliency Prediction [17.948179628551376]
  We propose a vision-transformer-based model for omnidirectional videos named SalViT360. We introduce a spherical geometry-aware self-attention mechanism that is capable of effective omnidirectional video understanding. Our approach is the first to employ tangent images for omnidirectional saliency prediction prediction, and our experimental results on three ODV saliency datasets demonstrate its effectiveness compared to the state-of-the-art.
  arXiv  Detail & Related papers  (2023-08-24T18:07:37Z)
• Panoramic Vision Transformer for Saliency Detection in 360{\deg} Videos [48.54829780502176]
  We present a new framework named Panoramic Vision Transformer (PAVER) We design the encoder using Vision Transformer with deformable convolution, which enables us to plug pretrained models from normal videos into our architecture without additional modules or finetuning. We demonstrate the utility of our saliency prediction model with the omnidirectional video quality assessment task in VQA-ODV, where we consistently improve performance without any form of supervision.
  arXiv  Detail & Related papers  (2022-09-19T12:23:34Z)
• Learning Omnidirectional Flow in 360-degree Video via Siamese Representation [11.421244426346389]
  This paper proposes the first perceptually natural-synthetic omnidirectional benchmark dataset with a 360-degree field of view, FLOW360. We present a novel Siamese representation Learning framework for Omnidirectional Flow (SLOF) Experiments verify the proposed framework's effectiveness and show up to 40% performance improvement over the state-of-the-art approaches.
  arXiv  Detail & Related papers  (2022-08-07T02:24:30Z)
• Deep 360$^\circ$ Optical Flow Estimation Based on Multi-Projection Fusion [10.603670927163002]
  This paper focuses on the 360$circ$ optical flow estimation using deep neural networks to support increasingly popular VR applications. We propose a novel multi-projection fusion framework that fuses the optical flow predicted by the models trained using different projection methods. We also build the first large-scale panoramic optical flow dataset to support the training of neural networks and the evaluation of panoramic optical flow estimation methods.
  arXiv  Detail & Related papers  (2022-07-27T16:48:32Z)
• Imposing Consistency for Optical Flow Estimation [73.53204596544472]
  Imposing consistency through proxy tasks has been shown to enhance data-driven learning. This paper introduces novel and effective consistency strategies for optical flow estimation.
  arXiv  Detail & Related papers  (2022-04-14T22:58:30Z)
• Learning optical flow from still images [53.295332513139925]
  We introduce a framework to generate accurate ground-truth optical flow annotations quickly and in large amounts from any readily available single real picture. We virtually move the camera in the reconstructed environment with known motion vectors and rotation angles. When trained with our data, state-of-the-art optical flow networks achieve superior generalization to unseen real data.
  arXiv  Detail & Related papers  (2021-04-08T17:59:58Z)
• Optical Flow Estimation from a Single Motion-blurred Image [66.2061278123057]
  Motion blur in an image may have practical interests in fundamental computer vision problems. We propose a novel framework to estimate optical flow from a single motion-blurred image in an end-to-end manner.
  arXiv  Detail & Related papers  (2021-03-04T12:45:18Z)
• Revisiting Optical Flow Estimation in 360 Videos [9.997208301312956]
  We design LiteFlowNet360 as a domain adaptation framework from perspective video domain to 360 video domain. We adapt it from simple kernel transformation techniques inspired by Kernel Transformer Network (KTN) to cope with inherent distortion in 360 videos. Experimental results show the promising results of 360 video optical flow estimation using the proposed novel architecture.
  arXiv  Detail & Related papers  (2020-10-15T22:22:21Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.