Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions

• URL: http://arxiv.org/abs/2208.12763v1
• Date: Fri, 26 Aug 2022 16:21:19 GMT
• Title: Leveraging Synthetic Data to Learn Video Stabilization Under Adverse Conditions
• Authors: Abdulrahman Kerim, Washington L. S. Ramos, Leandro Soriano Marcolino, Erickson R. Nascimento, Richard Jiang
• Abstract summary: We propose a synthetic-aware adverse weather robust algorithm for video stabilization. Our model generalizes well on real-world videos and does not require large-scale synthetic training data to converge.
• Score: 9.070630868911639
• License: http://creativecommons.org/licenses/by/4.0/
• Abstract: Video stabilization plays a central role to improve videos quality. However, despite the substantial progress made by these methods, they were, mainly, tested under standard weather and lighting conditions, and may perform poorly under adverse conditions. In this paper, we propose a synthetic-aware adverse weather robust algorithm for video stabilization that does not require real data and can be trained only on synthetic data. We also present Silver, a novel rendering engine to generate the required training data with an automatic ground-truth extraction procedure. Our approach uses our specially generated synthetic data for training an affine transformation matrix estimator avoiding the feature extraction issues faced by current methods. Additionally, since no video stabilization datasets under adverse conditions are available, we propose the novel VSAC105Real dataset for evaluation. We compare our method to five state-of-the-art video stabilization algorithms using two benchmarks. Our results show that current approaches perform poorly in at least one weather condition, and that, even training in a small dataset with synthetic data only, we achieve the best performance in terms of stability average score, distortion score, success rate, and average cropping ratio when considering all weather conditions. Hence, our video stabilization model generalizes well on real-world videos and does not require large-scale synthetic training data to converge.

Related papers

• BVI-RLV: A Fully Registered Dataset and Benchmarks for Low-Light Video Enhancement [56.97766265018334]
  This paper introduces a low-light video dataset, consisting of 40 scenes with various motion scenarios under two distinct low-lighting conditions. We provide fully registered ground truth data captured in normal light using a programmable motorized dolly and refine it via an image-based approach for pixel-wise frame alignment across different light levels. Our experimental results demonstrate the significance of fully registered video pairs for low-light video enhancement (LLVE) and the comprehensive evaluation shows that the models trained with our dataset outperform those trained with the existing datasets.
  arXiv  Detail & Related papers  (2024-07-03T22:41:49Z)
• Enhancing Indoor Temperature Forecasting through Synthetic Data in Low-Data Environments [42.8983261737774]
  We investigate the efficacy of data augmentation techniques leveraging SoTA AI-based methods for synthetic data generation. Inspired by practical and experimental motivations, we explore fusion strategies of real and synthetic data to improve forecasting models.
  arXiv  Detail & Related papers  (2024-06-07T12:36:31Z)
• Robust Monocular Depth Estimation under Challenging Conditions [81.57697198031975]
  State-of-the-art monocular depth estimation approaches are highly unreliable under challenging illumination and weather conditions. We tackle these safety-critical issues with md4all: a simple and effective solution that works reliably under both adverse and ideal conditions.
  arXiv  Detail & Related papers  (2023-08-18T17:59:01Z)
• Expanding Synthetic Real-World Degradations for Blind Video Super Resolution [3.474523163017713]
  Video super-resolution (VSR) techniques have drastically improved over the last few years and shown impressive performance on synthetic data. However, their performance on real-world video data suffers because of the complexity of real-world degradations and misaligned video frames. In this paper, we propose real-world degradations on synthetic training datasets.
  arXiv  Detail & Related papers  (2023-05-04T08:58:31Z)
• Towards Real-World Video Deblurring by Exploring Blur Formation Process [53.91239555063343]
  In recent years, deep learning-based approaches have achieved promising success on video deblurring task. The models trained on existing synthetic datasets still suffer from generalization problems over real-world blurry scenarios. We propose a novel realistic blur synthesis pipeline termed RAW-Blur by leveraging blur formation cues.
  arXiv  Detail & Related papers  (2022-08-28T09:24:52Z)
• Self-supervision versus synthetic datasets: which is the lesser evil in the context of video denoising? [11.0189148044343]
  Supervised training has led to state-of-the-art results in image and video denoising. It requires large datasets of noisy-clean pairs that are difficult to obtain. Some self-supervised frameworks have been proposed for training such denoising networks directly on the noisy data.
  arXiv  Detail & Related papers  (2022-04-25T08:17:36Z)
• Investigating Tradeoffs in Real-World Video Super-Resolution [90.81396836308085]
  Real-world video super-resolution (VSR) models are often trained with diverse degradations to improve generalizability. To alleviate the first tradeoff, we propose a degradation scheme that reduces up to 40% of training time without sacrificing performance. To facilitate fair comparisons, we propose the new VideoLQ dataset, which contains a large variety of real-world low-quality video sequences.
  arXiv  Detail & Related papers  (2021-11-24T18:58:21Z)
• Progressive residual learning for single image dehazing [57.651704852274825]
  A progressive residual learning strategy has been proposed to combine the physical model-free dehazing process with reformulated scattering model-based dehazing operations. The proposed method performs favorably against the state-of-the-art methods on public dehazing benchmarks with better model interpretability and adaptivity for complex data.
  arXiv  Detail & Related papers  (2021-03-14T16:54:44Z)
• Neural Re-rendering for Full-frame Video Stabilization [144.9918806873405]
  We present an algorithm for full-frame video stabilization by first estimating dense warp fields. Full-frame stabilized frames can then be synthesized by fusing warped contents from neighboring frames.
  arXiv  Detail & Related papers  (2021-02-11T18:59:45Z)

This list is automatically generated from the titles and abstracts of the papers in this site.

This site does not guarantee the quality of this site (including all information) and is not responsible for any consequences.